Content Negotiation and Representation Formats

One URI can stand for several byte sequences. That follows from the separation between a resource and its representation described in resource identity and URI design, and it is what makes negotiation necessary. The same order resource may be available as JSON, as XML, as a CSV export, or as a PDF invoice, and the same page may be available in three languages and two content codings. Content negotiation is the mechanism by which a client states which of those it can handle and the server picks one. It is also the mechanism most likely to be implemented halfway, producing responses that are correct for the requesting client and wrong for every cache between the client and the server.

The negotiation itself is straightforward: the client sends preferences, the server chooses, and the response records what was chosen. The complications come from three directions. Preferences are expressed as quality values with subtle precedence rules. A negotiated response must be marked so that shared caches do not serve it to clients with different preferences. And the choice of media type is a schema decision that constrains how the API can evolve.

Negotiation runs on independent axes

RFC 9110 defines negotiation over four request headers. Accept lists media types for the representation body. Accept-Encoding lists content codings, which are transformations applied to the representation for transfer. Accept-Language lists natural languages. Accept-Charset existed historically and is effectively obsolete in practice, because every widely used representation format now fixes its character encoding: JSON is defined only for UTF-8 by RFC 8259, and HTML and XML declare their encoding in the document.

The axes are independent, and a server that negotiates one but not the others is making an implicit promise about the rest. Compressing a response with gzip does not change its media type; the Content-Encoding header records the coding, and the underlying Content-Type still describes the decoded bytes. A response that declares Content-Type: application/json and then sends gzip bytes without Content-Encoding is unparseable, and this mistake survives testing because most HTTP client libraries transparently decompress only when the header is present.

Quality values and specificity pick the winner

Each element of an Accept header may carry a q parameter between 0 and 1, with at most three fractional digits. The value expresses relative preference within that header, not an absolute measure. A missing q means 1. The default when the header is absent is */*, and the default when a server cannot honour anything is to send its own preference anyway.

Selection is by most specific matching range, with quality used to rank candidates only after specificity has been resolved. The classic trap is the interaction of */* with a lower q value. In Accept: text/*;q=0.9, */*;q=0.1, a request for text/html matches both ranges; the more specific text/* wins with q=0.9 even though another server might reasonably read the wildcard as the intended preference. Conversely, q=0 is an explicit refusal: Accept: application/json;q=0 means JSON is not acceptable, not that it is least preferred. Servers that clamp or discard zero values will send a representation the client declared it cannot consume.

A second trap is parameter matching. Accept: application/json; charset=utf-8 matches a representation with the same type and parameter, and a range without parameters matches any parameters. The q parameter is not part of the media type; it belongs to the header grammar. Comparing media types as strings, including parameters, is a reliable way to get the wrong answer.

def parse_accept(header):
    ranges = []
    for part in header.split(","):
        fields = [f.strip() for f in part.split(";") if f.strip()]
        media = fields[0].lower()
        q = 1.0
        for f in fields[1:]:
            if f.startswith("q="):
                q = float(f[2:])
        ranges.append((media, q))
    # Higher q first; ties keep header order, which is a legitimate tie-break.
    return sorted(ranges, key=lambda r: -r[1])

Precedence between ranges of equal specificity is not defined by a total order, and servers differ in whether the first listed range or the highest quality wins. Clients should not depend on a tie-break, and servers should not construct responses whose correctness depends on one.

Vary is what makes negotiation cacheable

A shared cache keys responses by URI and method. If the response depends on Accept-Language, then a French client and an English client share a key, and whichever request arrives second receives the wrong language. The Vary header lists the request header fields that participated in selecting the representation, adding them to the cache key. Vary: Accept, Accept-Language tells the cache that two requests with different values of those fields may map to different responses.

Omitting Vary is a correctness bug with a security dimension, because it allows one client’s negotiated response to be served to another. Including too much is a performance bug: every distinct value becomes a separate cached variant, and a header with many possible values fragments the cache until the hit rate collapses. Vary: * means the response cannot be reused for any subsequent request, which is honest but equivalent to disabling caching.

There is a subtler problem with Vary on headers that intermediaries normalise. A cache that stores the raw Accept value treats application/json and application/json;q=1.0 as different variants, which is wasteful but safe. A cache that normalises aggressively may merge variants that the origin considers distinct. The specification permits either, and the practical guidance is to keep negotiated axes few and their value space small.

Media type design as an evolution tool

The Content-Type of a response is a schema identifier, and changing it is a compatibility event. The structured syntax suffix convention, defined in RFC 6839, appends +json or +xml to a vendor-specific type so that generic tooling can still recognise the underlying syntax. application/vnd.example.order+json tells a client that the body is JSON-shaped while the precise schema belongs to the vnd.example.order type.

In practice, very few APIs benefit from vendor media types. A client that must send Accept: application/vnd.example.order.v2+json is coupled to a versioning scheme just as tightly as one that calls /v2/orders, with less tooling support. The main legitimate use is when the same URI genuinely serves several incompatible schemas and a version in the path would be wrong, which is rare. Most APIs choose one media type and evolve the schema additively, as schema evolution discusses.

Error responses are the exception where a dedicated media type earns its place. RFC 9457 defines application/problem+json, a small object with type, title, status, detail, and instance members, plus extension members. Declaring it lets a client distinguish a machine-readable error body from an HTML error page produced by a proxy, and lets generic client code extract a human-readable message without knowing the API. The type member is a URI identifying the problem category, which is the extension point: a type of https://errors.example.com/insufficient-funds can carry a documentation page and a stable identifier that client code branches on.

Refusing, defaulting, and the 406 problem

A server that cannot satisfy any acceptable representation returns 406 Not Acceptable. Almost no production API does this. Browsers send broad Accept headers, but command-line clients, embedded HTTP libraries, and older proxies send narrow or malformed ones, and returning 406 turns a cosmetic mismatch into an outage. The common compromise is to ignore Accept for the primary representation and document a single supported type, which is honest as long as the API does not claim to negotiate.

Ignoring Accept has a cost when clients rely on it for export formats. A reporting endpoint that returns CSV for Accept: text/csv and JSON otherwise must set Vary: Accept, and must also handle the case where the client asks for CSV with a q lower than JSON’s. If the endpoint is heavily cached by a CDN, the variant space is small enough to be manageable. If it accepted arbitrary combinations of columns and formats, the variant space would not be.

Content coding negotiation

Accept-Encoding follows the same quality-value grammar with one addition: identity, the no-transform coding, is always acceptable unless explicitly refused with q=0. A client that sends Accept-Encoding: gzip;q=0, identity is saying it wants the bytes untouched, and a server that compresses anyway may break a client that is verifying a signature over the payload.

Compression is negotiated per response and recorded in Content-Encoding, and multiple codings can be applied in order, listed in the order applied. Applying gzip twice is legal and pointless, and it is a reliable way to defeat a scanner that inspects only the outermost layer. Transport compression also interacts with encryption: compressing a response that mixes attacker-controlled input with secrets is the basis of the BREACH class of attacks, which is why per-response compression of sensitive pages is often disabled even though the bandwidth cost is real.

Negotiation scales to a handful of axes with small value spaces. Beyond that, the variant count grows multiplicatively and the cache hit rate falls faster than the flexibility is worth. A reporting API that offers six formats, four locales, and free-form column selection is better served by an explicit job resource with parameters in the body, or by a format suffix in the path, than by an Accept header that encodes a combinatorial request. Header negotiation is a good fit for a small, closed set of representations and a poor fit for anything that behaves like a query.