Data Engineering · Cluster
Batch Processing
Partitioned jobs and shuffle mechanics.
- 01 Input Splits Determine Task Granularity in Batch Jobs How a batch framework derives splits from filesystem blocks, why split size sets the task count, and what happens when files are small or compression is not splittable.
- 02 How a Partitioner Maps Keys to Reduce Tasks Why the reduce task count is fixed before any record is read, what hash-modulo partitioning guarantees, and why changing the count remaps almost every key.
- 03 What a Shuffle Moves Between Map and Reduce Stages The mechanics of the map-side write, the network fetch, and the reduce-side merge: what actually crosses the wire, and why the stage boundary exists at all.
- 04 Pre-Aggregating Map Output with Combiners The associativity contract a combiner must satisfy, where the framework applies it, and how much shuffle volume it removes when key cardinality is low.
- 05 Sort-Based and Hash-Based Shuffle Implementations Two ways to build the map-side shuffle output, how each one trades sort CPU for file count, and why the file count grows with the reduce task count.
- 06 Hot Keys, Skew, and Load Imbalance in Hash Partitioning Why a partitioner that distributes keys evenly can still produce one task with sixty times the average work, and the static and sampled fixes that spread that load.
- 07 Map-Side Buffer Budgets and the Cost of Spilling Why the sort buffer is a fixed per-task budget, what the spill threshold implies for merge passes, and how much extra I/O a spill actually costs.
- 08 Co-Partitioned Datasets and Shuffle-Free Joins How persisting a hash or range partitioning in the table layout lets later joins read bucket against bucket, and the metadata mismatches that silently undo it.
- 09 Shuffle Fault Tolerance: Recomputing Lost Map Output Why map output is an unreplicated cache on local disk, what a lost segment costs downstream, and how lineage and shuffle services change the recovery path.
- 10 Adaptive Partitioning and Runtime Repartitioning How a shuffle writer reports per-partition sizes at the stage barrier, and how coalescing and splitting use that report to correct a partition count chosen before the data was read.