Multipart Upload: Choosing a Part Size That Survives a Retry

Why a single request cannot carry an object

A PUT is bounded because the service commits an object by writing one index record that names a set of immutable extents. A single request has to buffer, checksum, and replicate everything it carries before it can acknowledge, and the acknowledgment is what makes the write durable. Raise the bound and the amount of state a single in-flight request holds grows with it, along with the time a client spends with no partial progress recorded.

Multipart upload decomposes that commit into two phases. The client opens an upload, receiving an upload identifier. It then sends parts, each numbered in the range 1 to 10,000, each independently checksummed and independently acknowledged; an acknowledged part is durable and can be retried, re-sent, or ignored without affecting the others. Finally the client sends a completion request listing the part numbers and their checksums, and the service assembles the object from the stored parts in one metadata operation.

The published bounds define the arithmetic. A part is at least 5 MiB except the last one, which may be any size including zero. A part is at most 5 GiB. An object is at most 5 TiB. With 10,000 parts, the smallest part size that can express a maximum-size object is 5 TiB divided by 10,000, which is about 524 MiB. Nothing requires parts to be equal, but a client that wants a fixed part size must choose one in the range that both exceeds the 5 MiB floor and, multiplied by 10,000, covers the object.

The floor exists to protect the index, not the network

The 5 MiB minimum is not a throughput heuristic. Each part becomes an entry in the object’s extent list, and each entry costs metadata bytes in the index record that every read of the object must consult. A 100 GB object in 5 MiB parts is 20,480 parts, which exceeds the 10,000 limit; the limit is the real constraint and the floor is what prevents a client from reaching it with tiny parts. Below the floor, an object of modest size would produce an index with more entry bytes than payload, and every GET would pay to read it.

The practical consequence is that small objects must not use multipart at all. A 4 MiB object sent as one part violates the floor unless it is the only part; sent as a single PUT it is one index entry. The threshold where multipart begins to pay is where the object is large enough that a retry of the whole thing costs more than the overhead of several requests, which is a function of link reliability rather than of object size alone.

Part size as a retry-cost decision

The dominant reason to use multipart on a lossy link is that a failed part costs only that part. The expected bytes retransmitted for an object of size S split into parts of size p is roughly S/p times the probability that a part fails times p, which is to say it does not depend on p at all. What depends on p is the variance and the tail. With large parts, a single failure costs a long stall and a large retransmission; with small parts, failures are cheap but the number of requests and the metadata overhead rise.

There is a second, harder cost: a part must be buffered somewhere before it is sent, or read twice. Streaming a part directly from disk requires the sender to know its length and checksum in advance, so either the sender reads the range twice, once to hash and once to transmit, or it buffers the range in memory. Part size therefore sets the memory floor of the uploader, and a 512 MiB part size implies at least that much resident memory per concurrent part if checksums are computed before sending.

A planning function makes the constraints concrete:

MIB, GIB = 1 << 20, 1 << 30
MIN_PART, MAX_PART, MAX_PARTS = 5 * MIB, 5 * GIB, 10_000

def plan(size, target_parts=64):
    """Smallest part size within [5 MiB, 5 GiB] that keeps the count legal."""
    part = max(MIN_PART, min(MAX_PART, -(-size // target_parts)))
    part = max(part, -(-size // MAX_PARTS))   # grow if the count would overflow
    count = -(-size // part)
    assert count <= MAX_PARTS and part <= MAX_PART
    return part, count

for size in (100 * MIB, 1 << 30, 1 << 40, 5 << 40):
    part, count = plan(size)
    print(f"{size >> 20:>8} MiB -> {part >> 20:>5} MiB x {count}")

A 100 MiB object returns the 5 MiB floor and 20 parts; a 1 TiB object hits the 5 GiB per-part ceiling and needs 205 parts; a 5 TiB object stays at that ceiling and needs 1,024 parts, still well under the count limit. The last part is whatever remains, so it may be smaller than the rest.

Completion is a commit, and it can fail

The completion request names each part number and its checksum, in ascending order. The service validates that every named part exists, that the sizes are within bounds, that the checksums match, and that the numbers are contiguous from 1 with no gaps. A part that was uploaded but omitted from the list is discarded; parts not yet acknowledged are not waited for. Completion is therefore atomic in the sense that either the object appears with the full set of named parts or no object appears at all.

Two failure modes follow. First, a completion that fails validation leaves the upload open, with every part still billed. Second, a completion that succeeds but whose response is lost leaves the client uncertain: the object may exist. Retrying the completion with the same part list is generally safe because the operation is idempotent for an already-completed upload on the major services, but a client that instead restarts the upload produces a second complete object under the same key, and the last write wins.

The entity tag of a completed multipart object is not the digest of the object. It is derived from the digests of the individual parts, with a suffix giving the part count, which is why it can be compared against a locally computed digest only by recomputing the same construction over the same part boundaries. Two uploads of identical bytes with different part sizes produce different entity tags. Any workflow that deduplicates or verifies by entity tag must fix the part size as part of the contract, or use the whole-object checksum the service computes independently when one is requested.

Orphaned parts and the abort rule

An upload that is never completed leaves its parts stored and billed at the storage class of the bucket. Nothing expires them by default on the major services; the client is expected to abort. A lifecycle rule can abort incomplete multipart uploads after a stated number of days, and without such a rule a workload that crashes mid-upload repeatedly accumulates storage that no listing of objects will reveal, because the parts belong to an upload, not to a key.

This is the most common cost surprise in a large-object pipeline, and it compounds with versioning: a completed upload under a versioned bucket retains the previous version’s full footprint as well, as described in versioning and delete markers. The accounting question of what a stored byte costs over time, including the minimum-duration charges that apply to archival classes, is in lifecycle transitions and minimum storage durations.

Reading back what was written in parts

Because parts are aligned to known offsets, a completed multipart object is the natural target for parallel ranged reads: the same boundaries that made the upload resumable make the download splittable. The mechanics of issuing concurrent range requests and reassembling them, including how a partial response signals the range it returned, are covered in byte-range reads and parallel fetch. The symmetry is not accidental, since both operations exploit the fact that an object is an ordered list of immutable extents, but only the read side can be retried at arbitrary boundaries, since the write side’s boundaries are fixed at completion time and cannot be revised afterwards.