Deserialization as Re-entry Across the Trust Boundary

Every other boundary in this cluster is crossed by code the runtime chooses to run. Deserialization is the boundary crossed by bytes that decide, themselves, which code runs. A format that names types can be made to construct any type the runtime can reach, and the construction of an object is not a passive act: constructors run, setters run, and finalizers run later. That is why the parsing of untrusted input is a recurring root of runtime compromise.

The format determines the authority

There is a sharp division between formats that describe data and formats that describe objects.

JSON, per RFC 8259, describes data: objects, arrays, strings, numbers, booleans, null. A JSON parser produces a tree of a small number of generic types, and the application decides what to do with it. The parser has no mechanism to name an application class, so no input can cause the parser to construct one.

A format that carries type information behaves differently. Java serialization writes a stream of class descriptors and field values; the reader resolves each descriptor to a class by name, allocates an instance, and sets its fields. Python’s pickle is more direct still: it is a stack language whose opcodes include GLOBAL, which imports a named module and looks up a named attribute, and REDUCE, which calls a callable with arguments popped from the stack. A pickle stream is a program for constructing objects, and it can call any callable that is importable.

The pickle documentation states plainly that the module is not secure against erroneous or maliciously constructed data, and that unpickling data from an untrusted source is equivalent to allowing arbitrary code execution. That is not a defect in the implementation; it follows from the format’s semantics. A stream that can name a callable and call it is a general evaluation mechanism.

Gadget chains and the reachability problem

Formats in the middle ground, which resolve class names but do not call arbitrary callables, are still dangerous because of gadget chains. The mechanism is worth following precisely.

The attacker cannot inject a call to a dangerous function directly, because the format only constructs objects and sets fields. But the attacker can choose which classes to construct, and the classes available are every class on the application’s classpath, including library classes the application never intended to deserialize.

Some of those classes have the property that setting a field or invoking a setter causes a side effect. A class whose setter stores a value into a map that is later read by a lookup, a class that reads a field and treats it as a resource name, a class whose readObject performs a lookup or a load: each is a gadget. Chaining gadgets yields a path from “attacker chooses a class name and a field value” to “attacker’s code runs”.

The chain does not require any single class to be wrong. Each class behaves as designed when used as designed. The defect is that deserialization lets an attacker use it in a way the designer never considered, which is the definition of a gadget. This is why the fix is not to patch the gadget classes: the reachable set is the whole classpath, and it changes whenever a dependency is added.

The three containment strategies

The strategies differ in where they put the decision.

Allowlist the types. The reader is given the set of classes it may instantiate, and rejects any descriptor outside it. This is the only strategy that addresses the reachability problem directly, because it bounds the set of reachable classes to those the protocol actually uses. Its cost is maintenance: the list must track the protocol’s evolution, and a class added for a new feature is invisible to deserialization until the list is updated. The failure mode is a rejected legitimate message, which is loud.

Disable polymorphic reconstruction. If the format does not let the sender name a type, the reachability problem disappears. A protocol that carries a discriminator from a closed set, or that requires the receiver to know the expected type from context, has no gadget surface. This is the design of most well-specified wire formats: protobuf messages declare their fields in a schema, and the schema is fixed at build time, so a message cannot introduce a type the schema does not mention. The cost is expressiveness, since the sender cannot introduce new types without a schema change.

Isolate the parse. If neither of the above is possible, the deserialization runs in a context that limits what a gadget can reach. A separate process with a restricted syscall set, per syscall filtering, means a successful gadget chain gains the process’s authority rather than the application’s. This is a mitigation, not a fix: the attacker still executes code, and the containment is only as good as the process boundary.

Constructors are effects

The deep issue is that object construction is not a pure allocation. In languages where constructors can perform I/O, register themselves, start threads, or validate against external state, constructing an object is an observable action with side effects that the deserializer did not intend to authorize.

The classic shape: a class whose constructor opens a file given a name, or whose readObject/__setstate__ equivalent performs a lookup. Deserializing an instance of that class performs the operation, with arguments the attacker chose. The object is discarded immediately afterwards, but the effect has happened.

Formats that bypass constructors avoid this class of problem. A format that allocates the object without running its constructor, then sets fields directly, prevents constructor effects but creates a worse problem: an instance that has never been through its own invariants. A class whose constructor establishes an invariant, such as “the list is non-null” or “the index is within range”, can be given a state that violates it, and every later method on that object is operating on a lie. The bypass is what makes many gadget chains possible, because the object’s own defensive checks never ran.

Neither choice is safe by itself. The resolution in practice is to deserialize into data-only types and construct domain objects explicitly afterwards, so that construction happens under the application’s control with validated inputs. The intermediate representation has no behaviour, so it cannot be a gadget; the domain object’s constructor runs with values that have already been checked.

Depth, size, and the shape of the input

Even a format with no type information has resource hazards, and they are a distinct class from code execution.

Nesting depth is the first. A parser that recurses on nested arrays can be driven to exhaust the stack with a few thousand levels of nesting, and a stack overflow in a language without a guard is a crash at best and a memory corruption at worst. Parsers therefore impose an explicit depth limit, and the limit is a policy decision: deep enough for legitimate data, shallow enough that the parser’s own recursion cannot exceed its stack.

Total size is the second. A length prefix is an attacker-controlled integer, and a reader that trusts it allocates before reading. A four-byte length can request four gigabytes. The defences are to cap the declared length against the remaining input, to allocate incrementally as bytes arrive, and to apply a global limit on the message size before parsing begins.

Expansion is the third. Formats that support compression or references can produce output far larger than the input: a decompression bomb, or a document whose repeated references expand into a huge tree. The limit must apply to the expanded size, not the input size, and enforcing it requires the parser to count as it goes rather than to check afterwards.

A fourth, subtler hazard is duplicate keys and ambiguous encodings. If two representations of the same logical value are accepted, an attacker can choose which one a downstream component sees, and a validator that inspects one while the consumer reads the other has been bypassed. Strict parsers reject duplicates and reject non-canonical encodings, which costs compatibility with sloppy producers and buys the property that there is exactly one parse of a given byte string.

Where the boundary actually is

The trust boundary is not “the network” or “the file”. It is the point at which bytes become values the runtime will act on. A configuration file on disk, a message from a peer, a cached object, a cookie, an environment variable: each is a point where data enters and becomes authoritative. Deserialization is the general name for that transition, and the security question is always the same: what can the producer of those bytes cause the consumer to do?

The answer depends on the format’s expressiveness, and the design rule follows from it. It is the same question capability design asks about naming: whether an operation accepts a name for anything, or only a reference to something already granted. The less a format can say, the less an attacker can cause. A format that can only express data in a fixed schema is the safest and the least flexible; a format that can name types is more flexible and requires an allowlist; a format that can name and call code is a general execution mechanism and must be treated as such, which means never applying it to bytes from a source that is not already trusted to execute code.

The temptation to use a rich format for convenience, because it round-trips an object graph without writing conversion code, is the temptation to give the sender the ability to construct the receiver’s objects. That ability is authority, and it is granted by the choice of format, before any input arrives. The choice is therefore a security decision, and it belongs with the other boundary decisions in this cluster rather than with the serialization library’s ergonomics. The same reasoning applies to the reverse direction: a runtime that serializes its own objects into a form another process will deserialize is exporting a set of constructible types, and the exported set should be the protocol’s schema, not the application’s classpath.