Data Engineering · Cluster
Data Formats
Columnar layout, serialization, and schemas.
- 01 Columnar Layout and the Cost of Reading One Field How storing each field in its own contiguous run changes the bytes a scan touches, why homogeneous columns compress better, and what the layout gives up on point reads and updates.
- 02 Null Representation: Validity Bitmaps Against Sentinel Values How a columnar format records the absence of a value, why a separate bitmap costs one bit per value, and when an in-band sentinel breaks the type system.
- 03 Varints, Zigzag, and Prefix Encoding of Integers Why a variable-length integer encoding trades a length prefix for width, how zigzag folds signed values into the small-number space, and where the scheme degenerates to fixed width.
- 04 Inside a Parquet File: Row Groups, Pages, and the Footer How a Parquet file arranges column chunks, pages, and a trailing metadata block, why the footer is written last, and what the layout forces on readers and writers.
- 05 Dictionary Encoding and the Cardinality Cliff How replacing repeated values with ordinals shrinks a column, why the win collapses as distinct values grow, and what happens to a column chunk when the dictionary outgrows its budget.
- 06 Schema Evolution Without Rewriting Files Why field names cannot identify a column across versions, how numeric field identifiers make renames and reordering safe, and which schema changes remain illegal regardless of identifiers.
- 07 Column Statistics and the Page-Skipping Decision How min/max bounds, null counts, and bloom filters let a reader discard data without decoding it, and why clustering, sorting, and type all change how much they prune.
- 08 Repetition and Definition Levels in Nested Columns How two small integers per value encode the full structure of nested and repeated fields, what an empty list looks like on disk, and why the encoding stays independent of the record shape.
- 09 Arrow Buffer Layout and Where Zero-Copy Stops How an Arrow array is a handful of flat buffers plus a length, which conversions can reuse memory in place, and which schema and lifetime conditions force a copy.
- 10 Compression Blocks and the Loss of Splittability Why a compressor with a shared window makes a file unreadable from the middle, how block-structured codecs restore parallel reads, and what the block size trades away.