Contextual Keywords and Extending a Grammar in Place

Adding syntax to a language with existing users is constrained by a rule that has nothing to do with design taste: any program that was valid before must still be valid, and must mean the same thing. The cheapest way to satisfy that rule is to pick a keyword no existing program uses as an identifier. That is impossible in practice, because every short English word has been used as a variable name somewhere. The technique that resolves the conflict is to make the new keyword contextual: reserved only in the positions where the new syntax appears.

The cost of a hard keyword

A hard keyword is reserved everywhere. while in C cannot be a variable name, and a program that used it as one before the keyword existed stops compiling. That is why the list of reserved words in an established language grows slowly and why the words chosen tend to be either rare or already reserved.

C99 added inline and restrict as keywords, and code written before that used both as identifiers. A program in that position had to rename the identifiers, because once the standard reserves the spelling the plain name stops being available to the programmer. Compilers commonly provide a spelling with a reserved prefix, such as a double-underscore form, so that code can use the feature without spending the plain name, and the spelling is an extension rather than part of the language.

Java took the opposite route and kept a list of reserved words it does not use, including const and goto, so that future syntax could claim them without breaking anything. The list is a hedge: it costs programmers two unusable identifiers and buys the language a fixed budget of future keywords.

Reserving a word only in context

The alternative is to let the lexer emit an identifier and let the parser decide. A token is classified as a keyword when the grammar is in a state that expects one, and as an identifier otherwise. The classification depends on the parse position, so it cannot be done in the scanner, which sees one token at a time and no grammar state.

Python’s soft keywords are the clearest documented case. match, case, and _ are ordinary identifiers except in the specific positions where a match statement can begin, and the reference states that a soft keyword is treated as a keyword only in the relevant context. match remains usable as a function name, a variable, and an attribute; _ remains usable as a variable in most positions. The language gained pattern matching without taking three common names away from every existing program.

The implementation requires the parser to look ahead. When a statement begins with the identifier match, the parser must decide whether this is a match statement or an expression statement whose first token happens to be a variable named match. The decision is made by attempting the match-statement production and falling back, or by a bounded lookahead that distinguishes the two forms, such as whether the token following match can begin a subject expression and whether a colon eventually appears at the right depth.

That fallback is exactly the mechanism in Automatic Semicolon Insertion and Restricted Productions, where a parse failure triggers a repair. Here the repair is not insertion but reclassification: a failed attempt at the new production causes the parser to retry the old one. The two techniques share a cost, which is that the parser’s behaviour depends on how far it got before failing, and diagnostics can point at a token that the successful parse never reaches.

Context-sensitive words in C++

C++ uses several overlapping mechanisms, and the vocabulary is worth separating because the rules differ.

An identifier with special meaning is not a keyword at all. final and override are ordinary identifiers, and the grammar gives them meaning only in a virt-specifier position after a declarator. A class member named override is legal, and a variable named final is legal. The standard lists these separately from keywords precisely because they do not reserve the name.

A context-sensitive keyword is a token whose meaning depends on the grammatical context in a way the grammar itself encodes. The C++20 module syntax works this way: import and module were added to the list of identifiers with special meaning rather than to the keyword list, so they remain valid identifiers elsewhere and acquire their declaration meaning only at the start of a declaration. The restriction applies to a position rather than to the token, which is why a variable named module still compiles.

The third mechanism is the alternative token spelling, such as and for &&, which the standard provides so that keyboards without certain characters can write the operators. Those are keywords in their own right, and they are unavailable as macro names for the same reason. The committee adopted them as macro spellings in C95 and then made them keywords in C++, so code that defined the names as object-like macros works unchanged under the older rule.

Each mechanism has a different failure mode. An identifier with special meaning cannot be protected by a macro, since the standard forbids defining one with that spelling. A context-sensitive keyword can be ambiguous in a position where both readings parse. An alternative token is a keyword, so the collision it cannot prevent is with a name chosen in a language that treats the same spelling as an ordinary identifier.

The ambiguity that context buys

Every contextual keyword creates at least one string that two productions can derive, and the language needs a rule to choose. The rule is usually “the new syntax wins in the position where it is expected”, but the position must be identifiable without unbounded lookahead.

JavaScript’s async is a case where the rule is intricate. async is not a reserved word; it is an identifier that acquires meaning as a modifier before a function or an arrow function. async function f() {} is an async function declaration. async alone is a valid identifier, and async(x) is a call to a function named async. The specification resolves this with grammar productions that include the contextual keyword and a rule that a line terminator cannot appear between async and the following function. The restriction is the same family as the restricted productions described earlier, and it exists to keep the grammar deterministic: with the line break allowed, a parser that has read async and a line terminator could not decide by bounded lookahead whether the next token begins a function or an expression.

The general technique for managing this is to restrict the new production as tightly as possible. A contextual keyword should appear in exactly one grammatical position, with as much of the following context as possible fixed, so that the lookahead needed to disambiguate is bounded by a small constant. Languages that specify an unbounded lookahead requirement impose a cost on every parser, including the ones users write for tooling.

What it costs the tooling

A contextual keyword makes the grammar harder to use outside the compiler. Syntax highlighters, which are usually regular-expression based, cannot tell a match statement from a variable named match without a parser. Editors that highlight keywords by word list will colour the identifier form, which is wrong but harmless, or will fail to colour the keyword form, which is confusing.

Parsers generated from a grammar with contextual keywords need semantic predicates or a hand-written disambiguation layer, and not every parser generator supports them. A recursive-descent parser can implement the lookahead directly, which is one reason hand-written parsers remain common for languages with evolving syntax.

The specification also grows. Each contextual keyword needs a paragraph describing where it applies, and the paragraphs interact: a word that is soft in one version may become hard in the next, and the change is visible in the reference. Python’s reference separates soft keywords from keywords for exactly this reason, and the separation is a promise that the soft list can grow without breaking identifiers.

When to reserve instead

The contextual technique is not always the right answer. If a word is unlikely to be used as an identifier, reserving it is simpler and produces better diagnostics: a hard keyword is recognised by the scanner, so an error about a misplaced else is reported at the token rather than at the end of a failed production. Contextual keywords tend to produce errors that point at the wrong place, because the parser tried the new production first and failed somewhere inside it.

The second consideration is the size of the language’s user base. A young language can reserve words freely, since the cost is paid by code that does not exist yet. An established one cannot, and the contextual technique is the price of evolution. The third is the tooling ecosystem: a language with a published, parser-generator-friendly grammar benefits from hard keywords, because every tool can share the grammar.

The choice is visible in the shape of the resulting grammar, and it interacts with everything else in this cluster. A contextual keyword is a grammar-level ambiguity resolved by position, which is the same problem as The Dangling Else and How Grammars Resolve Ambiguity and the same solution family: a rule outside the production chooses between two valid derivations. The difference is that the dangling-else rule is fixed once grammars depend on it, while a contextual keyword’s rule is written as a grammar-internal condition and can be sharpened in a later version.

Desugaring Boundaries: Where Surface Syntax Ends is the other half of the same compatibility problem. New syntax can be introduced as a desugaring into existing forms, which requires no new grammar at all, or as a new production, which requires the keyword question to be settled. Choosing between the two is a decision about how much the language is willing to grow its grammar, and every contextual keyword is a vote for growing it carefully rather than not at all.