Arrow Buffer Layout and Where Zero-Copy Stops

An Arrow array is not a container of values in the sense a language’s list is. It is a length, a null count, a validity buffer, and one or more data buffers, all flat byte ranges with no per-element headers. That description is deliberately thin, and it is the whole reason two processes, or two language runtimes, can share one without serializing it: the representation is a byte layout with no pointers to relocate, so a buffer written by one can be mapped by another and read directly.

Zero-copy is the usual word for that sharing, and it is conditional in ways that are easy to overstate. A conversion is zero-copy only when the source memory, the target layout, the type, and the lifetime all permit reuse.

The buffers per type

A primitive array such as int32 has a validity bitmap of $\lceil n/8 \rceil$ bytes and a data buffer of $4n$ bytes. The buffers are padded rather than tightly sized: the specification recommends 64-byte alignment and padding the end of each buffer so that a slice’s data can be over-read by a vectorized kernel, and the IPC format requires at least 8-byte alignment for the body. A boolean array is the exception that proves the rule, storing values one bit each in a bitmap and therefore carrying no separate data buffer.

Variable-width types store two buffers: an offsets buffer of $n+1$ integers and a data buffer holding bytes. String uses 32-bit offsets and LargeString uses 64-bit, which caps the former at 2 GiB per array and lets the latter exceed it. The choice is not cosmetic: a 32-bit offset array halves the offset overhead of a string column but forces chunking on large data, and a reader that concatenates chunks into one large array may overflow it.

A list array is a validity bitmap, an offsets buffer, and a child array. The child array is itself a full array, possibly with its own validity and data buffers, and the list’s offsets index into it. This is structurally different from the Dremel encoding used in files: Arrow stores nesting as child arrays referenced by offsets, so an element can be addressed directly, and the structure is materialized in memory rather than reconstructed from levels. The trade is that the in-memory form carries more per-value overhead in the offsets, and the on-disk form carries level streams that reconstruct the same structure. Repetition and Definition Levels covers the file side of that comparison.

A struct array has no buffers of its own beyond validity; its fields are child arrays of the same length. A dictionary array has an indices buffer plus a dictionary, which is a separate array of the distinct values; the indices are integers of a declared width, and the dictionary is shared across arrays that reference the same values.

Slicing, and why offsets are cheap

An array can be sliced without copying. The slice adjusts the array’s offset field, which counts elements, not bytes, and recomputes the length; the underlying buffers stay where they are. For a primitive array this is free. For a variable-width array the slice must adjust the offsets buffer, or, in the more common implementation, keep the original offsets buffer and record an element offset so that the first value’s bytes begin at offsets[offset].

The catch is that a slice retains the whole underlying buffer. Slicing 100 rows from a 10-million-row array keeps 10 million rows of memory alive, because the buffer is referenced as a unit. A long-lived slice of a large array is therefore a memory leak in effect, and the fix is a compaction step that copies the slice into its own buffer, which is a copy, so the zero-copy property was only ever about the initial slice operation.

Alignment has a similar subtlety. A slice of a primitive array by an odd number of elements produces a data pointer that is no longer 64-byte aligned, and vectorized kernels that assume alignment must either handle the misalignment or copy. Implementations handle this by dispatching to unaligned loads, which is cheap on x86-64 and slower on some other architectures, or by copying the slice into an aligned buffer before the kernel runs.

The four conditions for zero-copy

Reuse of memory across representations requires all of the following.

Same physical layout. The source must already be in the target’s buffer arrangement. A row-major array of structs is not; a struct-of-arrays with the same field order and the same validity convention is. Converting Parquet pages into Arrow arrays is a decode, never a copy-free mapping, because pages are compressed and encoded and Arrow buffers are not.

Same type and width. An int32 array cannot be reinterpreted as int64 without a widening pass, and a float64 cannot be read as int64 at all, since the bit patterns mean different things. A string array cannot be read as a list of bytes without a change of logical type, and even that change is not free in every implementation because the two types differ in their metadata.

Same endianness. Arrow specifies little-endian byte order for its buffers. A big-endian producer must byte-swap, which is a copy. This is stated in the format rather than left to the platform, which is what allows a file written on one machine to be mapped on another.

Compatible lifetime. The reader must not outlive the buffer’s owner. A memory-mapped file keeps its pages valid only while the mapping exists; an allocator’s buffer is valid only until freed. Zero-copy sharing across a process boundary therefore requires an ownership protocol, which is why the IPC format exists: it carries the schema and the buffer descriptions, and the receiving side maps or reads the body and reconstructs arrays that point into it.

// Sharing an Arrow buffer without copying: the slice borrows the original.
fn head(buf: &[i32], n: usize) -> &[i32] {
    &buf[..n]                 // no allocation, no element copy
}

// A width change is not a reinterpretation and must allocate.
fn widen(buf: &[i32]) -> Vec<i64> {
    buf.iter().map(|&v| v as i64).collect()
}

The distinction in those two functions is the whole idea. Borrowing a subrange preserves the layout; changing the element width produces a new layout, and the copy is unavoidable.

The IPC boundary

Arrow’s IPC format serializes a schema and a sequence of record batches, each a set of arrays with a description of their buffers. The message metadata is a FlatBuffers table, chosen because it can be read without a parse pass and without allocation: the table’s fields are offsets from the table’s own position, so a reader computes addresses arithmetically. The body is the buffers, concatenated with padding to 8-byte boundaries and, for the recommended alignment, 64 bytes.

Two properties make the format usable for exchange rather than only for files. First, the body can be streamed: a record batch is self-contained, so a producer can write batches as they are produced and a consumer can process each as it arrives, with no footer and no global index. Second, a reader can construct arrays that point into the received body if the body is memory-mapped or otherwise retained, which is the zero-copy path; if the transport delivered the body as a copied message, the copy already happened and the arrays point into that copy, which is still zero-copy with respect to the array construction.

Compression is where IPC and zero-copy part ways. A record batch can be compressed as a unit, with each buffer compressed independently or the body compressed as one block. Once compressed, the buffers are not directly readable, so the consumer must decompress into new memory and the arrays point at the decompressed copy. Compression is a bandwidth trade against a copy, and the choice depends on whether the bottleneck is the network or the memory bus. The same trade appears at the file level in Compression Blocks and the Loss of Splittability.

What the layout gives up

The buffer model is excellent for scanning and poor for mutation. There is no room to insert a value into a primitive array without shifting the tail, and no room to change a variable-width value without rebuilding the offsets from that point onward. Arrow’s answer is to treat arrays as immutable and to build new ones for changes, which matches the columnar access pattern and conflicts with the row-at-a-time update pattern that a transactional store needs.

It also gives up self-description at the value level. An Arrow buffer has no per-value type tag and no per-value length, so a reader with the wrong schema produces plausible garbage rather than an error. The schema travels beside the buffers and is trusted, which is why schema validation is a separate step and why a mismatched schema in a shared-memory exchange is a correctness problem, not a performance one. That trust is the price of the flat layout, and it is the same price every format in this cluster pays in a different currency: the metadata is the contract, and the bytes only mean what the metadata says they mean.