RPC Interface Definitions and Wire Framing

A remote procedure call style interface starts from a different premise than a resource interface. Instead of naming things and applying methods to them, it names operations and describes their inputs and outputs. That premise leads to an interface definition language: a schema file from which client stubs and server skeletons are generated, so that neither side hand-writes serialisation code. The schema is the contract, and the bytes on the wire are derived from it.

The wire format matters because it determines what can change without breaking peers. A field renamed in the schema is invisible on the wire; a field renumbered is a different field entirely. This article follows the path from an interface definition through protobuf encoding and gRPC framing to the HTTP/2 stream that carries it, then contrasts the design with a textual envelope format.

The interface definition is the source of truth

A protobuf service definition declares methods, and each method declares a request and response message type. Code generation produces a typed client whose methods look local and a server interface whose methods must be implemented. Nothing in the generated code describes how to reach the peer: that is the transport’s job, and separating the two is what allows the same schema to run over gRPC, over a custom connection pool, or in-process.

The schema also fixes field identity. Each field has a name, a type, and a number. The name is for humans and generated code; the number is what appears on the wire. This is why the protobuf style guide reserves field numbers permanently once used, forbids reusing a deleted field number, and provides the reserved statement to record that decision in the file. The valid range for field numbers is 1 to 536,870,911, with 19,000 through 19,999 reserved for the implementation, and numbers 1 through 15 cost one byte in the encoded key while 16 through 2,047 cost two. Frequently set fields belong in the low range for that reason.

Proto3 removed the required keyword that proto2 had, on the grounds that a required field is a schema change that cannot be deployed safely: adding one to an existing message makes old producers non-conformant. Presence is still expressible. Scalar fields in proto3 have implicit presence and are omitted when equal to the default, while the optional keyword restores explicit presence tracking for scalars, and message-typed fields always track presence.

How one message becomes bytes

The encoding is a sequence of key-value pairs with no delimiters and no total length. Each field starts with a varint key whose value is (field_number << 3) | wire_type. Wire type 0 is a varint, 1 is 64-bit fixed, 2 is length-delimited, 5 is 32-bit fixed. Wire types 3 and 4 belonged to the deprecated group encoding in proto2 and are not produced by current encoders. Because the key packs the field number and the type together, a decoder can skip a field it does not recognise by reading only its length or its fixed width.

def read_varint(buf, pos):
    result = shift = 0
    while True:
        b = buf[pos]
        pos += 1
        result |= (b & 0x7F) << shift
        if not b & 0x80:
            return result, pos
        shift += 7

def decode_fields(buf):
    pos, fields = 0, []
    while pos < len(buf):
        key, pos = read_varint(buf, pos)
        number, wire = key >> 3, key & 0x7
        if wire == 0:                      # varint value
            value, pos = read_varint(buf, pos)
        elif wire == 2:                    # length-delimited value
            length, pos = read_varint(buf, pos)
            value, pos = buf[pos:pos + length], pos + length
        elif wire == 1:
            value, pos = buf[pos:pos + 8], pos + 8
        elif wire == 5:
            value, pos = buf[pos:pos + 4], pos + 4
        else:
            raise ValueError(f"unsupported wire type {wire}")
        fields.append((number, wire, value))
    return fields

Two properties of this format have operational consequences. First, there is no schema identifier in the payload and no field-name text, so a message decodes only against a schema the receiver already has. Second, absent fields are indistinguishable from fields set to the default for implicit-presence scalars, which means a client cannot tell “the server did not send this” from “the server sent zero”. Applications that need the distinction must use explicit presence, a wrapper message, or a sentinel outside the type’s value range.

Repeated fields appear as one key-value pair per element for scalar types, or as a single length-delimited blob when the packed option is set, which is the default for numeric repeated fields in proto3. Decoders must accept both forms regardless of how the encoder was configured, because a peer compiled from an older schema may still use the unpacked form.

Framing a stream of messages

A protobuf message has no length prefix of its own, so a stream of messages needs external framing. gRPC uses a five-byte prefix: one byte indicating whether the message body is compressed, followed by a four-byte big-endian unsigned length. The receiver reads the prefix, then reads exactly that many bytes, then repeats. Concatenating two encoded messages without a prefix produces a byte string that decodes as one message with duplicated or merged fields, which is a corruption that often passes shallow tests because the merged message is still syntactically valid.

Partial reads are the classic bug in hand-written framing code. A TCP read returns whatever is available, so a loop that assumes a single read yields a whole prefix or a whole body will fail intermittently under load and pass reliably on a loopback connection. The correct structure buffers until the required count arrives, as io.ReadFull does in Go or recv in a loop does in Python.

gRPC on HTTP/2

gRPC carries that framed stream inside HTTP/2. A call begins with a HEADERS frame containing :method: POST, a :path of /<service>/<method>, content-type: application/grpc, and the te: trailers header, which is mandatory because gRPC reports the call status in HTTP trailers rather than in the response headers. The request body is written as DATA frames, each containing zero or more length-prefixed messages. The server responds with HEADERS, then DATA for the response messages, then a final HEADERS frame with the END_STREAM flag carrying grpc-status and an optional grpc-message.

Status codes are not HTTP status codes. An application error such as a failed precondition arrives as HTTP 200 with grpc-status: 9 in the trailers, while a proxy that cannot parse the request returns an HTTP error with no gRPC status at all. Clients must therefore distinguish transport failures, HTTP-level failures, and gRPC statuses, or they will retry the wrong things. Deadlines travel in a grpc-timeout header whose value is a decimal number followed by a unit character, one of H, M, S, m, u, or n, and the server is expected to cancel the work when the deadline passes rather than merely return late.

HTTP/2 provides three things the transport needs. Multiplexed streams let many concurrent calls share one connection without head-of-line blocking at the HTTP level, HPACK compresses the repeated header sets, and per-stream flow control prevents a fast sender from overrunning a slow receiver. The flow-control window starts at 65,535 bytes per stream and is replenished by WINDOW_UPDATE frames. A server that reads messages slowly but keeps its connection window open will still stall senders at the connection level, which is why flow-control behaviour is a common source of throughput surprises.

Message size limits sit above all of this. Many gRPC implementations default to a 4 MiB maximum receive size per message, and exceeding it fails the call rather than truncating. Streaming large payloads as a sequence of smaller messages is the standard remedy.

JSON-RPC as the textual counterexample

JSON-RPC 2.0 sends a single JSON object with jsonrpc, method, params, and id members, and returns an object with result or error. It is transport-agnostic, has no interface definition language, and relies on the JSON grammar for framing, so a message boundary is whatever the transport provides, such as one HTTP request body or one WebSocket text frame. A request without an id is a notification and must not receive a response, which makes notifications fire-and-forget at the protocol level.

The trade is legibility for strictness. A JSON-RPC payload is readable in a log and can be produced with curl; it is also several times larger than the equivalent protobuf for small numeric records, and nothing in the protocol constrains the shape of params. Servers validate it themselves or not at all. Batches are arrays of requests, and the specification leaves ordering of responses unconstrained, so clients must match by id.

When the schema and the bytes disagree

Wire compatibility has rules that are narrower than intuition suggests. Adding a field with a new number is safe. Removing a field is safe if its number is reserved and never reused, because an old peer will simply skip the unknown field, but the removed field’s values are lost and reusing the number lets an old peer misinterpret unrelated data as the removed type. Changing a field from int32 to string is incompatible because the wire types differ: the first is a varint and the second is length-delimited, so a reader expecting one misparses the other. Widening int32 to int64 keeps the same wire type and is safe in both directions for values that fit, with the caveat that a 64-bit value read into a 32-bit field is truncated.

Unknown fields are preserved by many protobuf implementations and re-serialised on forward, which keeps data intact through intermediaries but can also resurrect fields a service believed it had removed. The rules that make evolution predictable are the same ones that make API versioning mostly a matter of discipline rather than mechanism, and the same rules explain why error reporting needs its own message type when a status enumeration is not enough.