Data Engineering

Columnar layout, serialization, and schemas.

1. Columnar Layout and the Cost of Reading One Field
2. Null Representation: Validity Bitmaps Against Sentinel Values
3. Varints, Zigzag, and Prefix Encoding of Integers
4. Inside a Parquet File: Row Groups, Pages, and the Footer
5. Dictionary Encoding and the Cardinality Cliff
6. Schema Evolution Without Rewriting Files

Partitioned jobs and shuffle mechanics.

1. Input Splits Determine Task Granularity in Batch Jobs
2. How a Partitioner Maps Keys to Reduce Tasks
3. What a Shuffle Moves Between Map and Reduce Stages
4. Pre-Aggregating Map Output with Combiners
5. Sort-Based and Hash-Based Shuffle Implementations
6. Hot Keys, Skew, and Load Imbalance in Hash Partitioning

Event time, watermarks, and state.

1. Event Time and Processing Time in a Stream Operator
2. Watermarks as Completeness Signals
3. Window Assignment and Trigger Semantics
4. Measuring Disorder to Size a Lateness Budget
5. Keyed State Layout and the Cost of Keeping It
6. Interval Joins Over Event Time

Connectors, exactly-once claims, and backfill.

1. Cursor Columns and Where the Next Incremental Read Starts
2. Log-Based Capture and the Write-Ahead Log as a Change Feed
3. Snapshot to Stream: Establishing a Position Without Losing Writes
4. Where the Checkpoint Goes: Offset Ordering and Two Failure Directions
5. What Exactly-Once Delivery Requires at Each Hop
6. Backfilling a Table While Its Stream Keeps Running

Dimensional design and slowly changing dimensions.

1. Grain: What One Row of a Fact Table Means
2. Star Schemas and the Cost of Normalizing a Dimension
3. Conformed Dimensions and the Bus Matrix
4. Type 2 Slowly Changing Dimensions: Versions, Validity and Lookups
5. Choosing Between SCD Types 1, 2, 3 and 6
6. Mini-Dimensions and Junk Dimensions: Narrowing a Wide Fact Table

Dependency graphs and retry semantics.

1. Topological Readiness in Workflow Dependency Graphs
2. Trigger Rules and Failure Propagation Across a Dependency Graph
3. Idempotent Task Boundaries and Atomic Output Publication
4. Retry Policy Design: Error Classification, Backoff, and Jitter
5. Durable Run State and Scheduler Reconciliation
6. Logical Dates, Interval Boundaries, and Catchup Semantics

Validation, lineage, and anomaly detection.

1. Where a Schema Contract Is Enforced, and What It Cannot Catch
2. Row Identity and Why Deduplication Needs a Key, Not a Comparison
3. Measuring Freshness From Event Time, Not Arrival Time
4. How Null Rates and Distinct Counts Detect a Broken Join
5. Partition Statistics as Validation Signals
6. Column-Level Lineage and the Cost of Tracking It

Table formats and open storage layers.

1. How a Table Format Replaces Directory Listing with a Metadata Tree
2. File Pruning from Per-File Min and Max Statistics
3. Atomic Commits and Snapshot Isolation Between Concurrent Writers
4. Partition Evolution Without Rewriting Existing Data
5. Why Columns Need Identifiers Rather Than Names
6. Delete Files and the Cost of Merge-on-Read