Option Parsing Contracts and the Ambiguities of Command-Line Syntax

An argument vector is a list of strings with no structure. Every convention that makes -v, --verbose, and -o file mean something is imposed by the program after main begins, and the conventions in use are not one rule but three overlapping ones. A program that parses arguments is choosing a contract, whether or not it says so, and the choice determines which command lines are expressible.

The grammar POSIX fixes

The POSIX utility conventions establish a small grammar that most Unix tools follow. Options begin with a single hyphen and consist of one alphanumeric character. An option that takes an argument consumes the rest of the same word, or the next word if the option character is the last in its word. Multiple options without arguments may be grouped after one hyphen. A double hyphen alone terminates option processing, and everything after it is an operand even if it begins with a hyphen.

Guideline 10, the -- terminator, is the part that makes the rest usable. Without it there is no way to pass a filename beginning with a hyphen, and such filenames are legal. Any parser that does not implement -- cannot be used reliably on arbitrary paths, and any test suite that never exercises -- is not testing the interface.

The POSIX model stops option processing at the first operand. In tool file -v, the -v is not an option; it is a second operand. This is the behaviour of a strictly conforming parser, and it is why ls and other classic utilities accept options only before operands in the strict reading. The model is simple and unambiguous, and it makes the parse depend on position in a way users routinely forget.

The loop that implements it is the same in every C program, and its shape explains the conventions better than prose does:

#include <unistd.h>
#include <stdio.h>

int main(int argc, char **argv) {
    int verbose = 0;
    const char *output = NULL;
    int opt;

    while ((opt = getopt(argc, argv, "vo:")) != -1) {
        switch (opt) {
        case 'v': verbose = 1; break;
        case 'o': output = optarg; break;   /* attached or next word */
        default:  return 2;                 /* getopt already printed */
        }
    }
    for (int i = optind; i < argc; i++)     /* the operands */
        printf("operand: %s\n", argv[i]);
    return 0;
}

optind marks the first operand once the loop ends; in the permuting model it is moved as operands are shuffled out of the way, which is why the value has to be read after the loop rather than during it.

Permutation and the GNU divergence

GNU getopt changed the default: it permutes the vector so that options may appear anywhere, and the operands are gathered at the end. Under this model tool file -v and tool -v file are equivalent, which matches what most users expect and what most modern documentation shows.

The divergence is visible in the environment. Setting POSIXLY_CORRECT in the environment makes GNU getopt stop at the first non-option, restoring the strict behaviour; a leading + or - in the option string changes the same behaviour programmatically. Two identical command lines can therefore parse differently depending on a variable the user set for unrelated reasons.

Permutation has a real cost. It makes the meaning of a word depend on the whole line rather than on its position, so a program cannot stream its arguments and must collect them all before acting. It also makes some command lines ambiguous that were previously clear: in tool -o -v file, is -v the argument of -o or a separate option? Under both models the answer is that -o consumes the next word, so the question answers itself, but only because option-arguments are greedy. The general lesson is that permutation is safe precisely because the consuming rule is local.

Subcommands change the contract again

Tools with subcommands such as git, docker, and cargo impose a two-level parse. The first level recognises a small set of words and possibly a few global options; the second level parses everything after the subcommand word under that subcommand’s own rules. The result is that global options are usually only accepted before the subcommand, and that --help has two different meanings depending on where it appears.

The design forces a decision that the POSIX grammar never had to make: whether the subcommand word may appear after options, and whether an option belonging to the subcommand is allowed before it. Implementations vary. The variation is not a bug in any one of them; it is the absence of a shared specification for the shape, and it is why a wrapper that forwards arguments to a subcommand has to know which words are global.

Optional option-arguments do not compose

An option argument can be required, absent, or optional, and the third category is where parsers lose the ability to be predictable. The POSIX utility conventions state that option-arguments should not be optional, so the grammar in practice offers two cases: an option either takes an argument or does not. GNU getopt_long adds optional_argument, and the only unambiguous way to supply the value is to attach it to the same word, as in --color=auto or -O2.

The reason is that an optional argument cannot consume the next word without changing the meaning of every following word. Given tool --color file, a parser that lets --color consume the next word turns a filename into a colour setting; a parser that does not leaves the option valueless. Both readings are defensible, which is exactly the defect. Implementations resolve it by requiring attachment, and users who write --color auto then get a valueless option plus an operand that happens to be the word auto.

The same trap exists in short options. -O with an optional argument must be written -O2; -O 2 passes 2 as an operand. A tool that documents -O [level] is documenting an interface that cannot be parsed unambiguously, and the ambiguity surfaces later as a misread filename.

Bundling, negative numbers, and the hyphen convention

Grouping -abc into three options is specified and widely implemented, but it interacts badly with options that take arguments. If -a takes an argument, -abc means -a bc, not -a -b -c, and the parser has to decide at the moment it reads a whether the rest of the word is an argument or more options. This is why a required argument can be attached without a space, and why an option string that declares a: changes the reading of every grouped word that starts with a.

The declaration character is what makes this unambiguous to the implementation:

Declaration	Meaning	-abc reads as
abc	no option takes an argument	-a -b -c
a:bc	a requires an argument	-a bc
a::bc	a has an optional argument	-a with value bc

Negative numbers are the other perennial case. sort -k2 -n is fine, but a program expecting a numeric operand cannot distinguish -5 from an option unless it stops option processing early or requires --. Tools that accept negative numbers as operands usually treat any word matching a numeric pattern as an operand before consulting the option table, which is a special case layered on top of the grammar rather than part of it.

A single hyphen is reserved by convention for standard input or standard output, and it is an operand, not an option. Programs that treat - as an option character break every pipeline that uses the convention. The related convention that an absent file operand means standard input is not specified anywhere and varies between tools, including between tools in the same suite.

Abbreviation and the cost of convenience

GNU getopt_long accepts any unambiguous prefix of a long option name. --verb matches --verbose if no other long option begins with those four letters. The feature reduces typing and makes scripts fragile in a specific way: adding a new long option whose name shares a prefix with an existing one can turn a previously valid abbreviation into an ambiguous one, and the failure appears in a caller that was not changed.

The alternative, exact matching only, is what most parsers outside the GNU tradition do. It costs a few keystrokes and removes an entire class of version-dependent breakage. Either choice is defensible; the indefensible position is to accept abbreviations and not document that adding options can break callers.

What the parser owes the caller

Three obligations follow from the grammar rather than from taste. Every option that can be mistaken for an operand must be reachable after --. Every error must produce a non-zero exit status, because callers branch on the status and not on the message, as covered in exit status conventions. And the parse must be complete before any side effect occurs, since a tool that deletes the first file and then discovers an unknown option has done damage it cannot undo.

The remaining freedom is where to put the boundary between options and operands, and that choice propagates into everything built on top of the tool. A wrapper that forwards arguments must preserve the vector exactly, which means quoting every parameter, as described in field splitting and quoting; otherwise the parser receives a different vector than the user typed and no amount of correctness inside the tool can recover the original intent.