Data Engineering · Cluster
Lakehouse Architecture
Table formats and open storage layers.
- 01 How a Table Format Replaces Directory Listing with a Metadata Tree Why a lakehouse table is defined by a chain of metadata files rather than by the contents of a prefix, and what each level of that chain costs a reader to open.
- 02 File Pruning from Per-File Min and Max Statistics How bounds recorded per column chunk let a scan skip whole files, why the pruning rate collapses on high-cardinality or unsorted columns, and how the statistics get into the manifest in the first place.
- 03 Atomic Commits and Snapshot Isolation Between Concurrent Writers What makes a commit atomic when the storage offers only single-object writes, how two writers racing on the same table are serialized, and which conflicts are detected only at commit time.
- 04 Partition Evolution Without Rewriting Existing Data How a table can change its partition granularity while old files keep their original layout, why the manifest must record the spec that produced each file, and what queries pay for the mixture.
- 05 Why Columns Need Identifiers Rather Than Names How field identifiers let a renamed or reordered column keep its meaning across old files, which type promotions the format permits, and why a name-based schema cannot express either.
- 06 Delete Files and the Cost of Merge-on-Read How a row-level delete is expressed without rewriting a data file, what a reader must do to apply it, and why position-based deletes are cheaper than equality-based ones.
- 07 Small File Accumulation and What Compaction Must Preserve Why frequent small commits degrade a table long after the commits stop, how a compaction job rewrites files without breaking snapshots, and which invariants it cannot violate.
- 08 Snapshot Expiry, Reference Counting, and Files That Nothing Points To Why deleting an old snapshot does not delete its files, how a sweeper decides which objects are unreferenced, and the two ways that decision goes wrong in opposite directions.
- 09 Catalogs, Table Identity, and What a Table Name Resolves To Why a table needs a registry separate from its files, how the catalog performs the pointer swap that makes a commit visible, and what breaks when the name and the location drift apart.
- 10 Deriving a Change Feed from Immutable Snapshots How a table format turns the difference between two snapshots into a stream of inserted and deleted rows, which changes it cannot recover, and why ordering must come from sequence numbers.