Tag
mapreduce
- Apr 23, 2026 What a Shuffle Moves Between Map and Reduce Stages The mechanics of the map-side write, the network fetch, and the reduce-side merge: what actually crosses the wire, and why the stage boundary exists at all. data engineering
- Jul 2, 2025 Sort-Based and Hash-Based Shuffle Implementations Two ways to build the map-side shuffle output, how each one trades sort CPU for file count, and why the file count grows with the reduce task count. data engineering
- Sep 24, 2024 How a Partitioner Maps Keys to Reduce Tasks Why the reduce task count is fixed before any record is read, what hash-modulo partitioning guarantees, and why changing the count remaps almost every key. data engineering
- Nov 4, 2022 Input Splits Determine Task Granularity in Batch Jobs How a batch framework derives splits from filesystem blocks, why split size sets the task count, and what happens when files are small or compression is not splittable. data engineering