Tag

shuffle

7 articles

Apr 23, 2026 What a Shuffle Moves Between Map and Reduce Stages The mechanics of the map-side write, the network fetch, and the reduce-side merge: what actually crosses the wire, and why the stage boundary exists at all. data engineering
Jul 2, 2025 Sort-Based and Hash-Based Shuffle Implementations Two ways to build the map-side shuffle output, how each one trades sort CPU for file count, and why the file count grows with the reduce task count. data engineering
Jun 11, 2025 Shuffle Fault Tolerance: Recomputing Lost Map Output Why map output is an unreplicated cache on local disk, what a lost segment costs downstream, and how lineage and shuffle services change the recovery path. data engineering
May 6, 2025 Map-Side Buffer Budgets and the Cost of Spilling Why the sort buffer is a fixed per-task budget, what the spill threshold implies for merge passes, and how much extra I/O a spill actually costs. data engineering
Sep 24, 2024 How a Partitioner Maps Keys to Reduce Tasks Why the reduce task count is fixed before any record is read, what hash-modulo partitioning guarantees, and why changing the count remaps almost every key. data engineering
May 14, 2024 Pre-Aggregating Map Output with Combiners The associativity contract a combiner must satisfy, where the framework applies it, and how much shuffle volume it removes when key cardinality is low. data engineering
Dec 26, 2021 Co-Partitioned Datasets and Shuffle-Free Joins How persisting a hash or range partitioning in the table layout lets later joins read bucket against bucket, and the metadata mismatches that silently undo it. data engineering