Repository-Scale Codemods and the Review Problem

A codemod is a program that rewrites source. Applied to one file it is an editor feature. Applied to ten thousand files it is a different engineering problem, because the output is a diff no human can read in full, and the ordinary control that makes refactoring safe, review, does not scale to it.

The techniques that make large-scale changes work are mostly about replacing review with something else that is equally strong, and being honest about which parts of the change no longer have a reviewer.

The arithmetic that forces mechanization

Consider a repository of 50,000 files, of which 2,000 contain a call to a function whose signature is changing. Each affected file needs between one and four edits. The result is roughly 4,000 hunks. A reviewer reading at 200 lines per minute, at an average hunk size of 10 lines, needs over three hours of continuous attention for the diff alone, and attention at that duration is not the same quality of attention as a five-minute review.

Two consequences follow, and they are in tension. The first is that the change must be mechanized, because hand-editing 2,000 files is slower and less consistent than any script. The second is that mechanized changes are precisely the ones where review degrades, because the reviewer sees repetition and stops reading individual hunks.

The resolution is to shift the burden of proof from the diff to the transformation. A reviewer who cannot verify 4,000 hunks can verify the rule that produced them, and can verify that the rule was applied to the right set of files. That is a much smaller object of review, and it is checkable by construction rather than by reading.

Writing the rule instead of the change

A codemod is usually expressed as a pattern plus a replacement, matched against a syntax tree. The pattern language matters, because the expressiveness of the matcher determines how much of the transformation can be described declaratively and how much must be hand-written per file.

# ast-grep rule: rewrite deprecated call to the new form, only when the
# second argument is the literal timeout the new API removed.
id: replace-fetch-with-retry
language: javascript
rule:
  pattern: fetchJson($URL, 30000)
fix: fetchJson($URL, { timeoutMs: 30000, retries: 2 })

The declarative form has three advantages over a script that parses and rewrites. The rule is readable by someone who did not write it. The match is on structure, so fetchJson( url, 30000 ) with different spacing matches the same node, while the string fetchJson($URL, 30000) inside a comment does not. And the rule can be run in a report-only mode, which enumerates matches without rewriting, so the size of the change is known before any file is touched.

The report-only run is the single most valuable step in the process. It answers three questions that determine whether the codemod is ready: how many sites match, whether the matches are the ones expected, and whether any match is in a file that should have been excluded, such as vendored code, generated code, or a test fixture that asserts on the old string.

Where declarative matching runs out

A pattern matches syntax, not semantics. Two situations defeat it.

The first is when the transformation depends on a type. Renaming a method on one receiver type but not another requires resolution, which requires a compiler front end with a semantic model rather than a parser. This is the reason the large-scale refactorings described in the literature are built on the language’s own front end: the ClangMR work rewrote C++ across a large codebase by running Clang’s parser and semantic analysis per translation unit and applying rewrites to the resulting AST, precisely because a purely syntactic match could not distinguish overloads.

The second is when the transformation requires a global property, such as “only rewrite call sites where the result is not used as a boolean,” which is a data-flow question about each site’s context. Some of this is expressible in a rule language with relational constraints; the rest is a hand-written analysis.

The practical division: put the shape of the change in the rule and the exceptions in a list of explicit file exclusions, each with a reason. An exclusion list with reasons is reviewable; a script with an if per special case is not.

Proving the change is mechanical

The property that lets a reviewer accept 4,000 unreviewed hunks is that each one is an instance of the same rule. Establishing that requires evidence, not assertion.

Idempotence. Running the codemod twice must produce the same output as running it once. A transformation that keeps rewriting its own output is not a rewrite of a pattern; it is a process with a fixed point it may not reach. Testing idempotence is cheap: run the tool, commit, run again, and require an empty diff.

Reversibility, where possible. Some codemods have an inverse codemod that restores the original. If the inverse exists, applying it to the rewritten tree should reproduce the original byte for byte, which is a strong check that no information was lost. This does not hold when the rewrite changes structure rather than names, and it does not hold when the rewrite normalizes formatting.

Coverage of the match set. The count of rewrites must equal the count of matches from the report-only run, minus the exclusions. A discrepancy means the tool matched something it could not rewrite, which is the case where a partial transformation leaves the codebase inconsistent.

Compilation after each shard. The strongest available check is that the result builds. A codemod that produces syntactically valid but semantically wrong code fails at the type checker, and the type checker runs over every file.

def check_idempotent(transform, sources):
    """A codemod must reach a fixed point in one pass."""
    for path, src in sources.items():
        once = transform(src)
        twice = transform(once)
        if once != twice:
            raise AssertionError(f"{path}: not idempotent")

The check is worth running on the report-only output too, since a rule that keeps matching its own output usually also over-matches the original.

None of these establish behavior preservation. They establish that the transformation is uniform, which is a different and weaker claim, and it is the strongest claim available at this scale.

Sharding for review and for bisection

A single commit touching 2,000 files is unmergeable in most workflows and unbisectable in all of them. Splitting it is not cosmetic; the split determines what can be diagnosed when something breaks.

Three sharding axes, each with a different property.

By directory or ownership boundary. Each shard has a reviewer who knows the code. This is the only axis that preserves review, and it is the one to prefer when the repository has clear ownership. The cost is coordination: the shards must land in an order that keeps the build green, which usually means the interface change lands first and the call-site shards follow.

By transformation. If the change has several independent parts, each part becomes a shard. This keeps each shard’s diff uniform, so the reviewer’s model of the rule stays valid across the whole diff.

By file count, arbitrarily. This is the fallback when neither of the above applies. It produces shards of roughly equal size with no semantic coherence, and its only virtue is that a bisect over the shards identifies the offending file range.

The ordering constraint that dominates all three: the codebase must build after every shard. That means the old and new forms coexist during the change, which is the expand phase of Expand, Migrate, Contract: Changing an Interface Without a Flag Day applied inside a repository instead of across a deploy boundary.

What the reviewer is actually checking

Given a shard, the reviewer’s job changes. Reading every hunk is not feasible and pretending to do it is worse than not doing it, because it produces approval without scrutiny. The reviewable objects are:

The rule, read once, for whether it encodes the intended change.

The report-only output over the shard, for whether the match set is the expected set.

The exclusion list, for whether each exclusion is justified.

A random sample of hunks, for whether the tool’s output matches the rule the reviewer read. Sampling does not prove correctness and it catches the case where the rule does something other than what its author described, which is the most common failure.

A representative file read in full, to check that the surrounding code still makes sense. A rewrite can be locally correct and globally wrong: a call site rewritten inside a function that no longer has the precondition the new API requires.

That last item is the class of defect that no amount of rule verification catches, because it is not a property of the rule. It is a property of each site’s context, and it is the reason large-scale changes are dangerous in a way that small ones are not: the per-site reasoning that a human would do is exactly what the tool omits.

Where this stops working

Codemods handle changes that are expressible as a rule over syntax with a bounded set of exceptions. They fail on changes where the correct edit depends on intent.

Renaming a concept is the canonical failure. A codemod can rename every occurrence of account to billing_account, and it cannot tell which occurrences refer to a different concept that happens to share the spelling. The result compiles and the domain model is now wrong in a way that no compiler will find.

Restructuring control flow is the second failure. A rule can move a statement into a helper; it cannot decide whether the helper should return early, and the decision differs per site.

The honest boundary is this: a codemod is safe when the transformation is local, the rule is total over the match set, and correctness of each site follows from the rule rather than from the site’s context. When any of those fails, the change is not a codemod. It is a series of individual refactorings that happen to be numerous, and each one needs the per-site reasoning of Extract Function and the Preconditions the Compiler Checks.