Schema Evolution and Wire Compatibility Across Formats

A JSON member is identified by its name, and the parser matches names as strings. A Protobuf field is identified by a number, and the parser matches numbers as integers in a tag. That single difference produces the whole discipline of binary schema evolution: renames are free, numbers are permanent, and a deleted field leaves a hole that must never be filled.

Nothing enforces the rule at the protocol level. A reused field number decodes without error and produces a value of the wrong type, or of the right type with the wrong meaning, in a message that validates against its schema. The only protection is a reserved list and a code review discipline, which is why schema evolution is a governance problem dressed as a serialisation problem.

Field numbers are the identity

A Protobuf tag combines a field number with a wire type, encoded as a varint. Field numbers 1 through 15 fit their tag in a single byte, which is why the documentation recommends them for the fields that appear on every message; numbers 16 through 2047 need two bytes. The valid range is 1 to 536,870,911, and the block 19,000 to 19,999 is reserved for the implementation and cannot be used.

Because the number is the identity, three rules follow immediately. Renaming a field changes nothing in the binary encoding and is safe there. Removing a field leaves its number available for accidental reuse, so the number should be marked reserved. Changing a field’s number is a delete plus an add, and any old data or old reader will misread it.

message Invoice {
  reserved 4, 6 to 8;             // former fields, never reuse these numbers
  reserved "discount_code";       // also reserve the old name

  string id = 1;
  int64 total_cents = 2;
  InvoiceStatus status = 3;
  repeated LineItem lines = 5;
}

Reserving the name matters for a different reason: it prevents a future editor from reintroducing the same name with a different type, which confuses the compiled output and the review alike. Reserving only the number leaves the name available for a semantically different field.

There is no equivalent identity mechanism in JSON and no protection at all in formats that key by name. A JSON field renamed is a JSON field removed from the perspective of a strict reader, which is why tolerant readers and additive changes are the only safe strategy there, as described in tolerant readers and unknown fields.

Wire types constrain retyping

Each field number is paired with a wire type, and the parser uses the wire type to decide how many bytes to consume. There are four in practice: varint for integers and booleans, 64-bit fixed for fixed64 and double, length-delimited for strings, bytes, embedded messages and packed repeated fields, and 32-bit fixed for fixed32 and float.

Retyping is safe only when the wire type is unchanged and the interpretation still makes sense. int32 to int64 is wire-compatible in both directions because both are varints, though a 64-bit value read by an int32 field is truncated to 32 bits. int32 to sint32 is not: sint32 applies zigzag encoding, so the same bytes decode to a different number. int32 to string is not: the reader expects a length-delimited payload and finds a varint, so it treats the field as unknown and discards it.

The rule that covers the common cases is that changes within the same wire type family are usually safe, and changes across families are never safe. The exception is embedded messages, which share the length-delimited wire type with strings and bytes. A field that was a string and becomes a message decodes without a protocol error and may produce an empty or garbled message, because the parser will attempt to interpret the string’s bytes as a nested message. This is one of the few cases where the wire format gives no warning at all.

Repeated fields have their own history. In proto3, repeated scalar numeric fields are packed by default, which changes the encoding from one tag per element to one length-delimited blob. Readers are required to accept both forms, so a change between packed and unpacked is wire-compatible in both directions, but only if both sides implement the acceptance rule.

Presence, defaults, and the field that cannot be told apart

Proto3 scalar fields have no presence: a field set to its default value is not serialised, and a reader cannot distinguish an omitted field from one explicitly set to zero. This breaks the same distinction that matters for patch semantics in JSON, and it is the reason for the optional keyword in proto3, which restores explicit presence tracking by wrapping the field in a synthetic oneof.

The distinction matters most during migration. A field added with a default value is invisible to old readers, which is the desired behaviour. A field whose zero value is meaningful is a trap: the reader treats “not set” and “set to zero” identically, so a caller cannot clear a value by sending zero, and a server cannot tell whether the caller intended to.

google.protobuf.FieldMask serves a related purpose for updates: it names the fields a request intends to change, so an omitted field is unambiguously untouched. Wrapper types such as Int32Value move the scalar inside a message, which makes presence detectable at the cost of an extra level of indirection on the wire and in code.

Required fields and the migrations around them

Proto2’s required is the strongest statement a schema can make and the worst one for evolution. A required field cannot be added to an existing message without breaking every producer that does not set it, and it cannot be removed without breaking every consumer that expects it. Proto3 removed the keyword entirely, which is the correct default for a format used across independently deployed services.

Where a field is genuinely mandatory, the enforcement belongs in application validation rather than in the schema. The schema describes what can be on the wire; the API describes what must be true. Conflating them means that a validation rule change requires a schema change and a coordinated deployment, which is precisely the coupling schema evolution exists to avoid.

The migration from proto2 to proto3 is mostly a set of presence and default changes, and it is not a drop-in change for messages that relied on required fields or on explicit default serialisation. The practical route is a parallel message type with a translation layer, deployed and retired through the procedure described in deprecation windows and removal gates.

Forward, backward, and full compatibility

The vocabulary is worth fixing because the words are used inconsistently.

Backward compatible means a new schema can read data written by the old schema. Forward compatible means the old schema can read data written by the new one. Full means both directions.

Adding an optional field is backward compatible, because new readers know the field and old data lacks it. It is forward compatible in the weak sense that an old reader skips the unknown field, but the value survives a decode and re-encode only if that reader preserves unknown fields. Proto3 implementations retain unknown fields, but early proto3 releases did not, and code that reserialises messages through a language runtime which discards unknowns will lose the value even in a nominally forward-compatible system.

That detail has an operational consequence: a proxy or gateway that decodes and re-encodes messages to inspect them silently strips fields it does not know. The team that adds a field sees it disappear whenever the traffic passes through the intermediary. This is the binary analogue of the JSON round-trip problem and it is harder to notice, because there is no visible payload to inspect.

Avro resolves by name, not by number

Avro takes the opposite approach and makes the reader and writer schemas explicit at decode time. A reader resolves a datum written under a writer schema against its own schema, matching fields by name, ignoring fields the reader does not declare, and filling fields the writer never wrote from the default declared in the reader’s schema.

The consequences invert several Protobuf rules. Renaming a field is breaking unless the reader schema declares an alias. Adding a field requires a default value in the reader schema. Reordering is free, because names carry the identity. Removing a field from the reader’s schema is safe by construction, since resolution ignores what the reader does not declare.

Because resolution needs both schemas, Avro data is normally stored or transmitted with its writer schema, or with a schema registry identifier. That is a real cost in bytes and in operational complexity, and it buys a property Protobuf does not have: a consumer can read historical data correctly with the schema that produced it, rather than with an accumulated set of numbering conventions.

The choice between the two is therefore about the lifetime of the data. Protobuf’s number-based identity suits request and response messages that exist briefly and are read by code compiled against a current schema. Avro’s name-based resolution suits stored records read years later by consumers that need the writer’s intent. A system that stores Protobuf messages indefinitely and expects to read them after many schema revisions is relying on reserved numbers and discipline where Avro would rely on the data itself.