Macro Hygiene and Identifier Capture

An expansion that inserts a variable binding is a small act of scope creation, and it happens inside someone else’s code. If the name chosen for that binding collides with a name the caller also uses, one of the two meanings silently displaces the other. This is not a textual problem and it does not go away when the macro operates on parsed trees; it is a problem about which binding an identifier refers to once two independently written pieces of code have been spliced together.

Two directions of the same collision

Consider a macro that evaluates a form while holding a lock. In a non-hygienic language, the natural implementation introduces a variable to hold the lock:

(defmacro with-lock (lock &body body)
  `(let ((acquired ,lock))
     (unwind-protect (progn ,@body)
       (release acquired))))

The expansion binds a variable named acquired. If the caller’s body happens to reference a variable of its own called acquired, that reference now resolves to the macro’s internal binding instead — the macro has captured the caller’s identifier. This is capture by the macro, and it is the case most often discussed.

The other direction is subtler. Suppose the macro’s body wants to call a function that the expansion site is expected to provide, and the caller has a local binding with the same name:

(defmacro with-timing (&body body)
  `(let ((start (now)))
     (prog1 (progn ,@body)
       (report (- (now) start)))))

If the caller has a local now that does something unrelated, the macro’s references to now are captured by the caller’s binding. The macro is captured by the caller. Both directions are failures of the same property: an identifier’s meaning should depend on the code that wrote it, not on the code it was textually merged with.

Worse than either is the fact that both are silent. The program compiles, and the mistake appears as a wrong value, an unbound variable at run time, or a variable that a garbage collector never frees because a closure the macro created still references it.

Gensyms: renaming by hand

The oldest mitigation is to give every binding the macro introduces a name no caller could have written. Common Lisp provides gensym, which returns a fresh uninterned symbol — a symbol with a print name but no home package, so it is not eq to any symbol the reader could produce:

(defmacro with-lock (lock &body body)
  (let ((acquired (gensym "ACQUIRED")))
    `(let ((,acquired ,lock))
       (unwind-protect (progn ,@body)
         (release ,acquired)))))

The uninterned symbol is guaranteed distinct from every symbol the source can contain, because the reader interns symbols by name in a package and an uninterned symbol is by construction outside that table. Printing it yields something like #:ACQUIRED417, which is not readable syntax; that is the point.

gensym solves capture by the macro for variables the macro introduces. It does not solve capture of the macro by the caller: a gensym renames the macro’s bindings but not its free references. A macro that needs a helper function must obtain it from somewhere the caller cannot shadow — a namespace with a reserved prefix, a module that the expansion refers to by a qualified name, or an explicitly passed environment. Each is a convention enforced by discipline, not by the expander.

The second cost is that gensym is a run-time allocation and a run-time name. Because the symbol is created during expansion, the same macro expanded twice produces two different symbols, which means two different compiled functions. Compilers cannot share code between them, debuggers show generated names, and any tool that compares two expansions for equality sees them as different.

Hygiene as a property of the expander

Hygienic macro systems make renaming automatic and, more importantly, bidirectional. The mechanism introduced by Clinger and Rees, and refined into syntax objects by Dybvig and colleagues, attaches to every identifier a record of the lexical context in which it was written. When the expander splices a template into an invocation, identifiers from the template carry the macro definition’s context; identifiers from the invocation carry the caller’s context.

Binding resolution then compares contexts rather than spellings. Two identifiers with the same spelling but different contexts are different identifiers, and a binding introduced by one does not capture a reference from the other. The effect is that the macro’s acquired and the caller’s acquired coexist in the same let body without interfering, exactly as if the macro had used a gensym — but with no gensym call and, critically, in both directions: the macro’s free reference to now also resolves in the macro’s context, so a caller’s local now does not capture it.

(define-syntax swap!
  (syntax-rules ()
    ((_ a b)
     (let ((tmp a))
       (set! a b)
       (set! b tmp)))))

With hygiene, (swap! tmp x) works: the tmp in the template is a different identifier from the caller’s tmp, even though both print the same. Under a purely textual expander the same invocation would be wrong, and the standard example of the failure is a swap that leaves both variables holding the same value.

The implementation detail that makes this work is that a syntax object is not a symbol but a compound value: the datum, plus a scope set. Every binding form adds a scope to the identifiers it binds and to the body it covers; resolution at the end of expansion compares the scope sets of the reference and of every candidate binding. A binding matches when its scope set is a subset of the reference’s; among the bindings that match, the reference resolves to the one with the largest scope set, so the most specific binding wins and shadowing is a consequence of the same rule. Adding a scope is a set insertion, and scope sets are compared by subset, which makes resolution a search over bindings rather than a lookup by name. That is the real cost: name resolution stops being a hash lookup on a symbol and becomes a comparison of scope sets, and the sets can grow with the depth of nesting.

Deliberate capture

Hygiene is not always wanted. An anaphoric macro intentionally binds an identifier that the caller’s body is expected to reference — it for the value just tested, or this for an implicit receiver. In a hygienic expander that binding is written by constructing an identifier whose context is the invocation’s rather than the template’s, which datum->syntax does by copying a caller-supplied syntax object’s context onto a new name. The capture then happens because the macro asked for it, using the same context machinery that otherwise prevents it.

The distinction matters for language design. A system with no hygiene cannot distinguish a macro that meant to capture from one that did so by accident, and the second is far more common. A hygienic system makes capture an explicit request, which is why the request exists as a separate construction rather than as a side effect of how the template was written.

Why most languages chose neither extreme

C’s preprocessor has no hygiene at all, which is tolerable only because the language’s macros introduce no bindings: a replacement list can expand to declarations, but there is no lexical scope for the expander to reason about, and the standard idiom for a macro that needs a temporary is to require the caller to supply it. That convention moves the burden onto the programmer and is responsible for a large share of the reserved-identifier conventions in C libraries.

Rust’s macro_rules! is hygienic for local variables but not for items. A let binding introduced inside a macro expansion gets a distinct syntax context, so it cannot capture a caller’s local, and a caller’s local cannot capture the macro’s. Items — functions, types, statics — resolve at the crate or module level by name, so two macros that both define a helper called new collide in the same module. This split follows from where the two kinds of name live: locals are resolved during expansion, items after it.

Procedural macros in Rust have no hygiene at all, because the macro receives and returns raw token streams and the compiler cannot tell which tokens came from the template and which from the invocation. A procedural macro that emits a let binding must use a name unlikely to collide, or accept the collision. The consequence is visible in the ecosystem’s naming conventions for generated identifiers, and it is one of the reasons derive macros tend to generate code inside a block or an impl where the caller’s locals are not in scope. The mechanics of that translation are covered in procedural macros over token streams.

The trade is now clear enough to state as a rule. Hygiene costs a richer identifier representation and a slower resolution algorithm, and it buys locality: the meaning of an identifier is fixed by the code that wrote it. Its absence costs nothing at expansion time and pushes the entire burden onto naming conventions, which fail silently. Systems that skip hygiene usually do so because their expansion is token-level and the expander has no scope information to work with at all — which is the same limitation described in token substitution and tree substitution, and the reason hygiene and tree-structured expansion tend to appear together.