Desugaring Boundaries: Where Surface Syntax Ends

Every language with more than a handful of constructs has a surface syntax and a smaller core that the surface translates into. The translation is usually called desugaring, and it is a design decision with consequences: it fixes what the construct means in corner cases, it determines which diagnostics a programmer sees, and it decides whether the construct can be redefined by a library.

The core as a specification device

A core language is a subset of the surface language that is expressive enough to define everything else, and simple enough to specify precisely. Scheme’s report defines a set of derived expression types, among them let, cond, and, or, and do, by giving each a translation into the core forms lambda, if, set!, and procedure application. The translation is part of the specification, so two implementations that both conform to it accept the same programs with the same meaning.

The core does not have to be the implementation’s internal representation. An implementation may compile let directly to a stack slot rather than to a lambda application, and the observable behaviour is the same. The core is a specification device: it says what the derived form means by showing an equivalent program in a smaller language. That is enough to settle ambiguities, and it is much less work than specifying each construct independently.

The pattern recurs in every language with a formal definition. A language may define for in terms of while, while in terms of conditional jumps, and conditional jumps in terms of the machine’s branch instruction, with each layer documented. The layers are not all equally visible to the programmer, and where the boundary sits is a design choice.

The for-loop as a rewrite

The most consequential desugaring in modern languages is the for loop. Its surface form suggests iteration over a collection; its core form is a protocol call.

In Rust, a for loop over an expression is defined as a call to IntoIterator::into_iter on the expression, followed by a loop that repeatedly calls next on the resulting iterator and binds the pattern to the item. The rewrite is documented and the names are part of the standard library’s public interface, so a type that implements the trait can be used in a for loop without the compiler knowing anything about it. The desugaring is the extension point.

Python’s for is defined over the iterator protocol: the expression is passed to iter(), and the loop calls next() until StopIteration is raised. The protocol is part of the language’s data model, and a class that defines __iter__ and __next__ participates. The loop body is a block, and the else clause runs when the loop completes without break, which is a piece of the surface form that has no direct core equivalent and must be specified separately.

JavaScript’s for...of uses the iteration protocol defined by Symbol.iterator, which returns an object with a next method producing { value, done }. The protocol is defined in the specification and is used by spread syntax, destructuring, and yield* as well, so the desugaring of for...of is one client of a shared mechanism.

In each case the desugaring is what makes the loop extensible. A while loop cannot be extended by a library, because its condition is a boolean expression the compiler evaluates directly. A for loop over a protocol can be extended by any type that implements the protocol, and the compiler needs no special case. That is the design payoff, and it is the reason languages with pattern-based iteration have largely displaced index-based loops in new code.

The cost is a layer of indirection. An index loop over an array compiles to a pointer increment and a bounds check; a protocol loop compiles to a virtual call per element unless the compiler deviates from the desugaring to specialise. The deviation is legal only if it is unobservable, and it is unobservable only when the protocol methods have no side effects the programmer can detect. Compilers that specialise typically do so under a rule that the iterator type is known and its methods are not overridden, which is a whole-program assumption in a language with separate compilation.

Async as a rewrite with a state machine

The async and await constructs are the clearest case of a desugaring that changes the execution model rather than merely the syntax. An async function is defined by a translation into a state machine: the body is split at each await into segments, the locals that live across an await become fields of the machine, and the machine implements a poll interface that resumes at the last suspension point.

The translation is not something a programmer would write, and its details are usually not part of the surface specification. What is specified is the observable behaviour: an async function returns a future, polling the future runs the body until the next suspension, and dropping the future cancels the computation at that point. The state machine is an implementation technique that satisfies the specification.

The boundary matters for two reasons. First, the set of locals that can live across an await is constrained by the machine’s representation. A local whose type is not Send cannot be held across a suspension if the future is then handed to an API that may move it to another thread, as a thread-pool executor does when it spawns it. The compiler reports this as a trait bound failure on the generated future type, not as an error at the local, which is why the diagnostics for async code are notoriously indirect. The desugaring leaked into the error message.

Second, the size of the future is determined by the largest set of locals live at any suspension point. That is a compile-time property of the translation, and it is why an async function’s future can be much larger than its arguments. Languages that box the future hide the size at the cost of an allocation; languages that keep it inline expose the size in the type.

Macros and hygiene

A hygienic macro system is a desugaring facility with a scoping rule. A macro expands to code in the caller’s context, but the identifiers it introduces are resolved in the macro’s definition context, so a caller’s binding cannot capture them and a macro’s binding cannot capture the caller’s names.

Scheme’s syntax-rules is the canonical example: a macro that expands to a let binding a temporary name is safe because the temporary is renamed to a fresh identifier that cannot collide. The renaming is not textual; it is a property of the syntax object, which carries the lexical context in which it was created.

The alternative, an unhygienic textual macro, is simpler to implement and produces the classic capture bug. A macro that binds tmp will shadow a caller’s tmp, and a macro that refers to a caller’s x will pick up a different x if the caller has rebound it. C’s preprocessor is unhygienic, and the conventions around macro naming, such as prefixing private names, exist to manage the hazard.

The design question is where expansion happens. A macro expanded before parsing can produce tokens that change the grammar, which makes the grammar context-dependent. A macro expanded after parsing operates on syntax trees, which keeps the grammar fixed and makes hygiene implementable. Languages that chose the latter pay for a more complex front end and get a macro system that cannot break the parser.

What the boundary hides

A desugaring is a promise that the surface form has no meaning beyond its translation. That promise is easiest to keep when the translation is total and the core is small. It is hard to keep when the surface form has an interaction the core cannot express.

The for...else clause in Python is an example: the core loop has no notion of “completed without break”, so the clause needs its own specification. Rust’s for loop must bind the iterator expression to a temporary before the first next call, because the naive rewrite would re-evaluate the expression on each iteration, and the specification states the single evaluation explicitly. JavaScript’s for...of closes the iterator on an abrupt completion of the loop body, which is a behaviour of the loop and not of the protocol call, and it must be specified as part of the loop rather than derived from the rewrite.

Each of these is a place where the boundary leaks. The surface form is not fully explained by the core, and the specification grows a clause that says “and also this”. The number of such clauses is a measure of how well the desugaring was chosen.

The leak is also visible in debuggers and profilers. A stack trace from desugared code shows frames the programmer did not write, and a breakpoint on a line inside a for loop may land in the protocol call. Compilers mitigate this with source maps and inlining metadata, which are additional machinery whose need follows directly from the desugaring.

Choosing the boundary

Three considerations decide where to put it. The first is extensibility: a construct defined by a protocol can be extended by users, and a construct defined by a built-in operation cannot. The second is diagnostic quality: a construct defined by a rewrite reports errors in terms of the rewrite unless the compiler adds special cases, and the special cases are what make error messages readable. The third is specification cost: a core with few forms and many derived forms is compact, while a core with many forms is easier to implement directly but harder to define.

Expression-Oriented Syntax and the Value of a Block is a case where the boundary is unusually clean: a block’s value is defined by a single rule, and every construct that produces a value inherits it. Operator Precedence Tables and Why They Are a Design Artifact is the opposite case: precedence is a property of the surface notation, and the translation into a parenthesised core is the step that makes the grouping explicit, so the core records the tree that the table produced rather than the table itself.

The practical rule for a language designer is to write the translation down, name the protocol it uses, and count the clauses that the translation does not cover. Those clauses are the real cost of the sugar, and they are what a reader of the specification will trip over first.