Inside a Parquet File: Row Groups, Pages, and the Footer

A Parquet file is a set of column chunks preceded by a magic number and followed by a metadata block whose length sits in the last eight bytes. A reader opens the file, seeks to the end, reads the footer length, reads the footer, and from that point knows exactly which byte ranges hold which fields for which rows. Nothing is discovered by scanning. This is the defining property of the format, and almost every other design choice follows from wanting the footer to be small and the byte ranges to be independent.

The three-level hierarchy

The format partitions rows into row groups, and each row group into column chunks, and each column chunk into pages. The hierarchy exists because each level answers a different question.

A row group is the unit of horizontal partitioning: it holds some number of consecutive rows, all columns present, and it is the granularity at which a reader can decide to skip data based on statistics. A column chunk is the unit of vertical partitioning: the values of one field for every row in the row group, stored contiguously, which is the columnar property from Columnar Layout and the Cost of Reading One Field applied within a bounded range. A page is the unit of encoding and compression: values inside a page share a single encoding and a single compression codec, and a page is the smallest thing a reader can decode.

The sizes are conventional rather than mandated. Writers commonly target row groups of a few tens to a few hundreds of megabytes of uncompressed data, and pages of around one megabyte. A very small row group multiplies metadata and defeats compression, since each page restarts its dictionary and its encoder state. A very large one limits the granularity of skipping and increases the memory a single-threaded reader must hold, and it makes the file hostile to splittable parallel reads across a cluster, since a row group cannot be divided among readers. A default of 128 MB, as used by the original Java implementation, is a compromise between those forces rather than a requirement of the specification, and other writers default to different values.

Pages and what each type carries

Pages are typed. A data page carries encoded values, either the definition and repetition levels plus the values, or the values alone when the column is required and not repeated. A dictionary page carries the dictionary for the column chunk and must precede the data pages that reference it. The specification also defines an index page type, but the fine-grained skipping structures in current use are not pages at all: the column index and the offset index are separate blocks written after the data pages and located through the footer.

Each page is compressed independently and prefixed with a page header carrying the uncompressed and compressed byte counts, the encoding, and the definition and repetition level encodings. The header is itself encoded in a compact Thrift structure, so reading a page header is a small parse rather than a fixed-size struct read. The independence of pages is what allows a reader to fetch and decompress a single page without touching its neighbours, and what allows a writer to stream pages as it fills them.

Within a data page the values are encoded with one of a small set of encodings: plain, dictionary, run-length, or one of the delta variants. The encoding applies to the whole page, so a page is the unit at which an encoder commits. A column whose values change character halfway through a row group, sorted in one half and random in the other, gets a page-level decision but not a value-level one.

The footer and its contents

The footer holds a schema tree, the row group list with per-column-chunk metadata, and a key-value map for arbitrary annotations. Per column chunk, the metadata records the file offset of the chunk, its total compressed and uncompressed sizes, the path in the schema, the codec, and per-page offsets. Per column, statistics record the minimum and maximum values, the null count, and optionally a distinct count.

Two details matter for performance. First, the statistics are optional and typed: a writer that omits them produces a valid file that cannot be skipped, and a reader must treat a missing minimum as unknown rather than as a bound. Second, min/max comparisons follow the column’s logical type, not its physical type, so a byte-array column declared as a string compares by unsigned byte order while the same physical bytes declared as a signed integer compare numerically. A reader that applies the wrong comparison silently skips the wrong row groups, which produces wrong answers rather than slow ones.

The footer is written after all data, because it must contain the offsets and statistics of everything written before it. This ordering has consequences: the writer must either buffer metadata for the whole file in memory or maintain a spill structure, and the file cannot be made readable until the last page is flushed. A crashed writer leaves a file with data pages and no valid footer, which is indistinguishable from an empty file to a conforming reader; there is no partial-recovery path in the base format.

Footer size grows with the product of columns, row groups, and pages, and it is the part of the file every reader must fetch before it can do anything. A hundred-column schema split into a thousand row groups produces a hundred thousand column-chunk descriptors; at a few hundred bytes each, the footer runs to tens of megabytes, and a reader that wants one column still pays to parse all of it. The scaling is why writers limit row group count on wide tables and why some systems store the footer in a separate sidecar so that a query touching one field can read only the descriptors for that field. It is also why the format allows a reader to stop parsing early: the Thrift structures are length-prefixed, so a reader that has found the schema and the first row group can skip the rest of the metadata without decoding it.

# Reading the footer without a full parse of the data.
import struct

MAGIC = b"PAR1"

def footer_range(path):
    with open(path, "rb") as f:
        f.seek(-8, 2)
        tail = f.read(8)
        if tail[4:] != MAGIC:
            raise ValueError("not a parquet file")
        (length,) = struct.unpack("<I", tail[:4])
        end = f.seek(0, 2) - 8
        return end - length, length

The eight trailing bytes are four bytes of little-endian footer length and four bytes of magic. A reader that validates the magic at both the start and the end rejects truncated files cheaply, which is the reason the format repeats the magic at the head.

Column chunks versus row groups as the unit of parallelism

A reader splitting a file across workers has two choices: assign row groups, or assign column chunks. Row-group assignment is the simpler contract, because each worker reconstructs whole records and no coordination is needed beyond a list of byte ranges; it is also what most query engines do, and it is why a file with a single row group is read by a single worker no matter how many cores are available.

Column-chunk assignment splits a row group across workers by field, which suits a query projecting a subset of columns and aggregating per column, but produces partial records that must be combined. The combination is cheap for aggregation and expensive for anything needing row alignment, so engines typically split by row group and then parallelize within a group by column when the projection allows it.

Neither scheme can split a page. Pages are the atomic unit of decoding, which sets the lower bound on read granularity: a query selecting one row still decompresses the page containing it. Writers that expect point lookups therefore keep pages modest and add the optional page index, which records per-page min/max so a reader can locate the page without decoding the others.

Where the format pushes work onto the writer

Several properties of the layout are obligations on the writer rather than guarantees to the reader. Dictionary pages must be emitted before the data pages that reference them, so a writer that discovers late that dictionary encoding is unprofitable must either restart the chunk or accept the wasted dictionary. Statistics must be computed per chunk and per page, which means the writer cannot be a pure pass-through: it must decode enough to compare values under the column’s logical type. And the row group boundary is the writer’s choice, which means two writers with different buffering policies produce files that behave differently under the same query, a fact that matters when comparing scan costs across datasets.

The format also leaves compression at page granularity, which is the subject of Compression Blocks and the Loss of Splittability, and it leaves nested data to the level encoding described in Repetition and Definition Levels. Both are places where the simple three-level story acquires a second layer of detail.