Procedural Macros and the Token Stream Contract

A macro that runs arbitrary code is a compiler extension, and a compiler extension needs an interface. The interface chosen by Rust and by several other systems is deliberately minimal: the macro is a function from a token stream to a token stream, declared in a crate that the compiler builds and links separately, and called by the compiler during expansion. Everything the macro can do follows from that signature, and so does everything it cannot do.

The shape of the boundary

A procedural macro is a function in a separate crate, marked with an attribute that names its kind. It receives a TokenStream — a sequence of tokens where each token is an identifier, a punctuation character, a literal, or a delimited group — and returns a TokenStream. The compiler parses the returned tokens as whatever the invocation position requires.

use proc_macro::TokenStream;

#[proc_macro]
pub fn make_answer(input: TokenStream) -> TokenStream {
    let n: u32 = input
        .to_string()
        .trim()
        .parse()
        .expect("expected an integer literal");
    format!("fn answer() -> u32 {{ {}u32 }}", n)
        .parse()
        .expect("the formatted text is a valid item")
}

The conversion through to_string and back is not an implementation detail of this example; it is the honest description of the interface. The macro sees the invocation’s tokens as text-like data and produces output as text-like data. It has no access to the compiler’s syntax tree, no access to name resolution, no access to type information, and no access to the surrounding item.

The separation is a deliberate isolation boundary, and it exists for three reasons.

The first is compilation order. A procedural macro is a compiled artifact that the compiler loads and calls. If it could query the compiler’s type information, the compiler would have to be reentrant across a call into a dynamically loaded library, and the macro would observe compiler state mid-pass. Keeping the interface to tokens means the call happens at a well-defined point, with no borrow of compiler state live across it.

The second is versioning. A token stream is a stable, small data type. A syntax tree API is not: the compiler’s internal representation changes between releases, and exposing it would freeze those changes or break every macro on every upgrade. The token interface is the part that can be promised.

The third is trust. A macro is arbitrary code running inside the compiler’s process. The narrower the interface, the less it can corrupt. A macro can still panic, and a panic is reported as a macro failure rather than crashing the compiler, but it cannot hold a reference into the compiler’s arenas.

What the macro has to reconstruct

Because the interface is tokens, the macro must parse. The ecosystem’s answer is a parser library that the macro author uses to define a grammar for the expected input:

use syn::{parse_macro_input, DeriveInput};

#[proc_macro_derive(Describe)]
pub fn describe(input: TokenStream) -> TokenStream {
    let ast = parse_macro_input!(input as DeriveInput);
    let name = &ast.ident;
    // ... build output from ast ...
}

The parse is the macro’s own, which means the diagnostics are the macro’s own too. When the input does not match the expected grammar, the error is produced by the macro’s parser, and reporting it usefully requires constructing a span — a reference to a byte range in the original source — so the compiler can underline the offending tokens. Spans are carried by the tokens themselves, which is why the interface preserves them even though it discards structure: a token knows where it came from, and a group knows the span of its delimiters.

The span is the only channel through which a macro can attach information to source locations. There is no API for “warn about this type parameter”, only “emit a diagnostic whose span is this token’s span”. A macro that wants to point at a specific field must have kept that field’s identifier token, which is why parser libraries preserve spans on every node.

Error reporting degrades in a specific way as a result. An error inside macro-generated code is reported at a span the macro chose, which may be the invocation site, may be a token from the input, or may be a call-site span deliberately attached to generated tokens. The compiler cannot explain what the generated code was trying to do, because it never saw the macro’s intent — only its output.

Expansion order and the visibility problem

Procedural macros expand in a fixed order relative to other passes, and that order determines what they can see. Attribute macros and derive macros run before name resolution completes, so a macro cannot ask whether a type named in its input exists. Function-like macros expand during parsing of the item that contains them, which means their output participates in the parse of the enclosing item.

The ordering produces a class of failure that is easy to hit and hard to diagnose: a macro that generates an item and a macro that consumes that item must expand in the right order. If the consumer expands first, the item it expects does not exist yet, and the error is “cannot find” rather than “expansion order”. The workaround is to make the consumer an attribute on the item the producer generates, which forces the ordering by nesting rather than by declaration.

Derive macros have a further restriction: they receive only the item they are attached to, not the rest of the module. A derive that needs to know about a sibling type cannot see it, and the standard solution is to require the user to list the siblings in an attribute argument, which the derive reads from the token stream. This is why derive attributes so often carry a list of names that the macro could in principle have discovered for itself if it had module visibility.

Cost in the build graph

A procedural macro is compiled for the host, not for the target. Cross-compiling a crate that uses a procedural macro therefore requires building the macro crate for the build machine as well as the target, and the macro’s dependencies — a parser library, usually — are compiled for the host too. This is the reason a build’s dependency graph for a cross-compiled target contains two parallel copies of a substantial part of the tree.

Incremental compilation is affected in a more subtle way. The compiler tracks dependencies between items to decide what to re-expand after an edit. A macro’s output depends on its input tokens, which the compiler can track, and on anything else the macro read — environment variables, files, network state — which it cannot. A macro that reads a file to generate code produces output that the dependency graph does not model, so an edit to that file does not trigger re-expansion. The convention is to declare such dependencies through the build system, and the failure mode when that is omitted is stale generated code that survives a rebuild.

Why not a full syntax API

Scala 3 takes the other position: macros are typed, quoted expressions, and a macro can inspect the types of its arguments and return a typed tree. The interface is the compiler’s own AST, exposed through a stable quoted representation, and the macro is written in the same language it is manipulating.

inline def power(inline x: Double, n: Int): Double =
  ${ powerImpl('x, Expr(n)) }

The ' quote and ${ splice are the staging markers: 'x is a reference to the argument’s tree, and ${ ... } inserts the tree that the macro body returns. A splice’s operand is an expression of type Expr[T] at the stage in which the macro runs, and the surrounding code expects a T, which the compiler checks against the tree rather than by re-parsing text. Because the trees are typed, the macro can match on the type expression it was given and reject an argument of the wrong shape, and Expr.valueOrAbort extracts the constant value when the argument is a literal, and the non-throwing value returns an option for the cases that are not known at expansion time.

The price is the one the token interface was designed to avoid: the quoted representation must remain stable across compiler versions, which constrains what the compiler’s internals can change, and the macro must be compiled by the same compiler version that will run it. Rust’s token interface accepts weaker diagnostics in exchange for a macro that is an ordinary crate with an ordinary dependency graph.

The practical boundary

The interface shape determines what a macro can be trusted to do. A token macro can rearrange syntax, generate items from a declaration, and embed computed values. It cannot perform type-directed decisions, cannot see other items, and cannot produce a diagnostic that explains a type error in terms of the macro’s intent. Those limits are not incidental to the design; they are what makes the macro loadable as a library and cacheable as a build artifact.

The one thing the token interface does preserve is position, and position is enough for a large class of tools. A derive macro can attach the original field’s span to a generated error, so the message points at the field even though the code that produced the message did not exist in the source. The same preservation is what allows a macro to generate code that a formatter or an IDE can map back to the invocation, which is why the span-carrying token, rather than the tree, is the interface’s real content.

Where a macro needs to traverse a type definition and generate code per field, the token stream is sufficient, and the pattern is common enough to have a name and a standard set of pitfalls; that is the subject of derive macros and structural recursion. Where the need is to run code at compile time and get a value rather than a token sequence, the token interface is the wrong tool, and compile-time function evaluation provides the other one.