Streaming APIs: Framing, Backpressure, and Resumption

A request-response exchange has an obvious end: the body is finite, its length is either declared or discoverable at close, and the client knows when it has everything. A stream removes that guarantee. Messages arrive over time, the connection may last for hours, and either side may disappear without a clean close. Everything that distinguishes a working streaming API from a demo is a consequence of that: message framing that survives partial reads, a way to tell a stalled consumer from an idle one, and a resumption contract that does not silently drop events.

Three transports dominate, and they are not interchangeable. Server-sent events use ordinary HTTP with a text framing convention and a standard resumption header. WebSocket upgrades the connection to a frame-based protocol with bidirectional messages and almost no application semantics. gRPC streams run inside HTTP/2 and inherit its flow control. Each makes a different part of the problem easy and leaves another part to the application.

Framing has to survive a byte stream

TCP delivers bytes, not messages. A stream format must define where one message ends and the next begins, and it must do so without relying on read boundaries. A read may return half a message, three messages, or one byte; a loop that assumes otherwise works on a loopback connection and fails under load.

Length-prefixed framing, as used by gRPC, makes the boundary explicit: a fixed-size header states how many bytes follow, and the reader buffers until it has them. Delimiter framing, as used by server-sent events, uses a byte sequence that cannot appear in the payload, and the reader buffers until it finds the delimiter. Both require the same discipline in the reader: accumulate into a buffer, extract complete messages, and carry the remainder forward. Delimiter framing additionally requires an escaping rule if the delimiter can occur in data, which is why SSE payloads are line-oriented and cannot contain a raw blank line inside one event.

Text formats pay a second cost: encoding. SSE is defined over UTF-8, and a multi-byte character split across two network reads is not an error, it is an incomplete buffer. Decoding each chunk independently produces mojibake at the boundaries. The fix is to decode incrementally with a streaming decoder, or to work on bytes until a complete event is assembled.

Server-sent events and the resume token

An SSE response has the media type text/event-stream and a body of events separated by a blank line. Each event is a set of field: value lines. data contributes a payload line, and multiple data lines are joined with newlines. event names the type. id sets the last event identifier, which the client stores and sends back as Last-Event-ID when it reconnects. retry suggests a reconnection delay in milliseconds. A line beginning with a colon is a comment, and is the standard heartbeat, since any bytes keep intermediaries from considering the connection idle.

res.writeHead(200, {
  'Content-Type': 'text/event-stream',
  'Cache-Control': 'no-cache',
  'Connection': 'keep-alive',
});

let seq = 0;
const timer = setInterval(() => {
  const event = { type: 'tick', seq: ++seq, at: Date.now() };
  // Two newlines terminate the event; id enables Last-Event-ID resume.
  res.write(`id: ${seq}\ndata: ${JSON.stringify(event)}\n\n`);
}, 1000);

req.on('close', () => clearInterval(timer));

The browser client reconnects automatically, which is the feature that makes SSE attractive: EventSource retries after a dropped connection without application code, and sends the last id it saw. The server then has to decide what “resume” means. If it keeps a replay buffer keyed by event id, it can send the events the client missed. If it does not, the client resumes at the current position and has a gap it cannot detect on its own.

That gap is the central design decision. A resume token that is only a sequence number is a promise the server can keep only if it retains the events. A token that encodes a position in a durable log, such as a log offset, lets the server read forward from storage. A token that encodes a snapshot plus a sequence number requires the client to reconcile. When the retention window is shorter than a plausible disconnect, the server must signal the gap explicitly, with an event that tells the client to refetch state, rather than silently continuing.

SSE is one-directional. Client-to-server messages need a separate request, which is often fine: the subscription is a long GET, and commands are ordinary POSTs. The limitation is real for interactive applications, and it is the main reason to reach for WebSocket.

WebSocket defines a transport, not a protocol

A WebSocket connection begins as an HTTP request with Upgrade: websocket, a Sec-WebSocket-Key, and a version. The server answers 101 Switching Protocols, and from then on the connection carries WebSocket frames rather than HTTP messages. The opening handshake includes an Origin header, which servers should validate, because the protocol has no same-origin policy of its own and a browser will happily open a socket to any host that accepts it.

Framing is binary and compact. Each frame has a two-byte header at minimum: a FIN bit, three reserved bits, a four-bit opcode, a mask bit, and a seven-bit length. Lengths of 126 and 127 escape to 16-bit and 64-bit fields. Client-to-server frames must be masked with a key carried in the frame, a measure against cache-poisoning attacks through misbehaving proxies rather than a security boundary. Opcodes distinguish continuation, text, binary, close, ping, and pong. Text frames must contain valid UTF-8, and an invalid sequence is a protocol error that closes the connection with code 1007.

Everything above the frame layer is application-defined. There is no message type, no request identifier, no subscription semantics, and no resumption. A dropped WebSocket is simply gone; the client reconnects, and whatever happened in between is the application’s problem. Ping and pong frames exist, and a server that does not answer pings will see clients give up, but browsers do not expose ping to JavaScript, so keepalive in a browser application is usually an application-level message. Close codes carry a little meaning: 1000 for a normal closure, 1001 for going away, 1008 for a policy violation, 1011 for an internal error, and 1006 as a local indication that the connection closed without a close frame, which a peer never sends.

The consequence is that a WebSocket API needs its own protocol design: a message envelope with a type and a correlation identifier, sequence numbers for ordering, an acknowledgement scheme for delivery, and a resume mechanism keyed by a server-assigned cursor. That is a substantial amount of design that SSE gets for free, in one direction at least.

gRPC streams are HTTP/2 streams

A gRPC method is unary, server-streaming, client-streaming, or bidirectional, and the difference is which side may send more than one message. Every message in either direction carries the same five-byte prefix described in RPC framing: one compression flag byte and a four-byte length. The call’s status still arrives in trailers, after the last DATA frame, so a stream that ends without trailers is a broken stream rather than a successful one.

Cancellation is a first-class operation because it is an HTTP/2 reset. A client that calls cancel sends RST_STREAM with the CANCEL error code, and the server’s handler sees the context end; work that is not derived from that context continues to run. Deadlines work the same way: the grpc-timeout header sets a deadline that the server should propagate into its own calls. Half-close is separate from cancel. A client-streaming call ends its request by sending END_STREAM while still expecting a response, which is how the server distinguishes “done sending” from “gone away”. Server-streaming handlers that write into a full window block inside the send call, and that blocking is the backpressure signal on this transport.

Backpressure separates a slow consumer from an idle one

A producer that can generate events faster than a consumer can read them has three options: buffer without bound, block the producer, or drop. Buffering without bound converts a slow consumer into a memory leak, and a handful of stalled clients can exhaust a server. Blocking propagates the slowness back to the source; with a shared source, one slow consumer stalls everyone. Dropping loses data, and whether that is acceptable depends on the stream’s semantics.

Transport-level flow control is the first line. HTTP/2 gives each stream a window, initially 65,535 bytes, replenished by WINDOW_UPDATE frames as the receiver consumes data, and a connection-level window on top. HTTP/1.1 relies on TCP receive windows, which the kernel adjusts, so a slow reader eventually stops the sender at the socket. WebSocket runs over TCP with no additional flow control, which means a server that writes faster than the socket drains accumulates data in its own userspace buffers and in the kernel’s send buffer, both of which are finite and neither of which is visible to the application.

Above the transport, an application-level queue with a bound is the mechanism that makes the policy explicit. A per-connection queue of a few hundred messages, with a metric for depth and a rule for what happens when it fills, turns an invisible problem into a visible one. The rule can be disconnection with a code that tells the client to resynchronise, coalescing of state updates so that only the latest value matters, or dropping the oldest events for a feed where recency dominates. The choice depends on whether the stream is a sequence of facts or a view of current state, and the two need different resumption semantics.

Reconnection is a state problem

Every long-lived connection breaks. Load balancers impose idle timeouts, NAT tables expire, mobile devices change networks, and processes restart. A stream that cannot resume is a stream that loses data on every deployment, so resumption is not an enhancement.

The mechanism is a cursor plus a retention policy. The server assigns each event a monotonic identifier scoped to the subscription, retains events for a bounded window, and on reconnect reads from the cursor. The retention window sets the maximum tolerable disconnect, and it should be longer than the client’s reconnect backoff plus the time a deployment takes. When the cursor is older than the retained window, the server must return an explicit “too far behind” signal, and the client must refetch a snapshot and restart from its position. Silent truncation is the worst outcome, because the client believes it has a complete history.

Delivery is at-least-once by construction, since a server that crashes after sending but before recording the cursor will replay. Consumers therefore need to be idempotent with respect to event identifiers, or the protocol needs acknowledgements so that the server advances the cursor only on confirmation. Acknowledgements cost a round trip and a per-client cursor, and they are the right choice when the events drive side effects that cannot be repeated.

There is also an ordering constraint that interacts with resumption. Events must be delivered in the order they were assigned identifiers, or the client cannot tell whether a gap is real. A server that fans out from multiple partitions and interleaves the results loses that property; the usual remedy is a per-partition sequence with an explicit merge rule, or a single ordered log per subscription.

What kills idle connections

Idle timeouts are the most common operational surprise. Proxies and load balancers close connections that have carried no bytes for some configured period, often a minute or two, and the close can be silent from the application’s perspective. A heartbeat at an interval well below the shortest expected idle timeout keeps the connection alive and gives both sides evidence that the peer is still there. For SSE, a comment line is enough. For WebSocket, a ping frame or an application-level message. For gRPC, HTTP/2 keepalive pings, which some servers reject unless configured to permit them.

Response buffering is the second. A reverse proxy that buffers the response body before forwarding will hold every event until the response ends, which for a stream is never. The symptom is an API that works locally and delivers nothing once a proxy sits in front of it. The fix is at the proxy layer: disable buffering for the stream route and avoid transforming compression in a way that delays flushes. Compression itself interacts badly with streaming, since a compressor may hold bytes until it has enough to fill a block, and flushing after each event defeats most of the compression benefit.

The third is half-open connections. A TCP connection whose peer has vanished stays open on the local side until something times out, which can be minutes or longer. Application heartbeats detect the absence of pongs, and a server that tracks last-seen time per connection can close and release resources deliberately rather than waiting for the kernel. The same timer that keeps the connection alive is the one that detects its death, which is why the heartbeat interval and the liveness threshold should be designed together rather than chosen independently.

The error model of a stream is different from that of a request: a mid-stream failure is a close with a reason, not a status code, and the client’s recovery logic has to distinguish “retry from the cursor” from “the subscription is gone”. The distinction belongs in the close reason, where both sides can see it.