Execution Counts and the Hidden State of a Notebook Kernel

A notebook file is a list of cells in document order. The outputs stored next to those cells were produced by some other order, and the small integers in the margin are the only surviving record of it. A file whose counts read 1, 2, 3, 7, 4, 5, 6 is not corrupt; it is a truthful transcript of a session that ran the seventh cell before the fourth. Reading such a file top to bottom and treating the result as the computation that produced the displayed numbers is the most common way a computational result becomes irreproducible without anyone noticing.

The mechanism behind that gap is not subtle once the two processes involved are separated: a kernel that holds an interpreter namespace, and a front end that holds a file.

Two processes sharing one document

The front end is an editor. It parses the notebook, renders cells, and sends code to a kernel over a socket. The kernel is a separate operating system process, usually a Python interpreter but potentially Julia, R, or anything else speaking the same protocol, running a loop that receives execution requests and returns results.

The wire protocol defines several channels. An execute_request travels on the shell channel and carries the code string plus flags governing whether output should be silent, whether errors should abort, and whether the front end wants a execute_reply on the same channel. The kernel emits busy and idle status messages on the IOPub broadcast channel around the work, and during execution it publishes stream messages for writes to standard output and standard error, display_data for rich output, execute_result for the value of the final expression, and error with an exception name, value, and traceback. Interrupts arrive on the control channel, which stays responsive while the shell channel is occupied.

Each executed code cell consumes one increment of a counter the kernel owns. The kernel reports the resulting integer in the execute_input message, in the execute_result message, and in the execute_reply. The front end stores it in the cell’s execution_count field. In the notebook format version 4, that field is an integer or null; null means the cell has never been executed in the session that wrote the file.

The counter is per kernel session. It begins at zero, is incremented before the code runs, and resets when the kernel restarts. Nothing about it is derived from the position of the cell in the document, which is why the two orderings can diverge so easily.

An increasing integer is a weak ordering witness

Because the counter only increases within a session, the sequence of counts read in document order reports where that order disagrees with execution order. A cell with a lower count than its predecessor ran earlier. A gap in the sequence — 1, 2, 4 — means one execution is missing from the file: either the cell that consumed count 3 was deleted afterwards, or its source was overwritten by an edit and the cell re-executed later, or the work happened in a cell that no longer exists at all.

Deletion does not renumber anything. Removing a cell leaves every other count untouched, because the numbers belong to executions rather than to positions. This is the property that makes a notebook useful as a log and useless as a program: the log records that something ran, not what ran.

A count is also overwritten rather than appended. Running the same cell three times leaves the last output and the last count in the file, not three entries. The stored document is therefore a snapshot of the final state of each surviving cell, and it contains no trace of the intermediate executions whose outputs were displayed, read, and used to decide what to do next. That intermediate reasoning is usually the part a reader most needs.

Two further properties of the counter are worth stating because they are often assumed away. A cell that raises an exception still consumes a count: the increment happens before execution, so a traceback in the file does not indicate a skipped execution. And counts are not comparable across sessions or machines; they are meaningful only relative to the kernel that issued them, and a restarted kernel starts again at one no matter how many executions preceded it.

The namespace is the real state of the computation

The document holds source text and rendered output. The computation holds a dictionary of names bound to objects, and that dictionary is what the next cell sees. Everything a cell does to the namespace persists until the kernel dies or the name is rebound, whether or not the cell that did it is still in the file.

The consequences are ordinary and severe. A cell deleted from the document leaves its variables defined; later cells continue to use them, and a fresh run of the visible document raises NameError at the point of first use. A cell that mutates an array in place — x[0] = 0, x.sort(), x.resize(...) — leaves the mutated object behind, so re-running an earlier cell that appeared to define x does not restore the original values. A cell that rebinds a builtin name, shadowing sum or list or range, changes the meaning of every later cell without changing a character of their source.

Module state survives for the same reason. Python caches imported modules in sys.modules, so editing a module file on disk and re-importing it in a long-lived kernel returns the cached module object rather than the edited code. Mutable module-level data — a configured logger, a cached lookup table, a global random generator that has already been advanced — keeps whatever the earlier cell left in it. Warnings filters set with warnings.simplefilter stay set. The working directory changed by os.chdir stays changed, and every later relative path resolves against it. Figures opened by a plotting library remain registered in its global state and can be drawn into by later cells, which is why a plot occasionally reappears in an unrelated cell’s output.

None of this is visible in the file. The document can be internally consistent, every count in order, every output plausible, and still describe a computation that cannot be repeated from its own text.

Restart and run all as the narrowest defensible claim

The one property that can be checked mechanically is the correspondence between document order and execution order. After a kernel restart, a run of every cell from top to bottom produces counts 1 through n in document order, with no gaps and no nulls. That invariant is cheap to test on a saved file, and it is the minimum condition under which the file can be read as a script.

The check is a few lines of JSON traversal, with no notebook library required.

import json, sys

nb = json.load(open(sys.argv[1]))
code = [(i, cell.get("execution_count"))
        for i, cell in enumerate(nb["cells"])
        if cell["cell_type"] == "code"]
ran = [(i, n) for i, n in code if n is not None]
print("code cells:", len(code), "executed:", len(ran))
print("gaps at:", [n for n, want in zip([c for _, c in ran], range(1, len(ran) + 1)) if n != want])
print("inversions:", [(a, b) for a, b in zip(ran, ran[1:]) if b[1] < a[1]])

A clean result from this scan says only that the visible cells were executed once each, in order, in some session. It does not say that the session was the one that produced the stored outputs, because the outputs are attached to cells rather than to executions and no timestamp ties the two together. It also does not constrain what happened outside the notebook: packages installed in a terminal, environment variables exported in a shell, data files written by a separate script, a GPU configured with a particular library build.

Passing the scan is necessary and far from sufficient. The environment the kernel ran in has to be recorded separately, which is the subject of Lockfiles, Hash Pinning, and Reconstructed Environments, and the notebook’s own limitations as a container are covered in What a Notebook File Stores and What It Omits. Even a fully pinned environment leaves the question of which inputs were read, which Content-Addressed Provenance Graphs for Derived Outputs addresses by hashing them.

Where the counter stops being informative

Several common arrangements defeat the visible-order test entirely.

A notebook that executes work through a helper script — %run experiment.py, or a subprocess call — records only the invocation. The count is consumed by the invocation, and the script’s own state changes, module imports, and file writes leave no cell-level trace. Two notebooks can appear identical at the cell level while running completely different code.

Parallel workers introduced from a notebook complicate the counter rather than the ordering. Work dispatched to a process pool executes in code that was never a cell, and its results arrive through a channel with no count attached. If the pool was created after the kernel started and inherited state, results can depend on which worker received which task, and nothing in the document records the assignment.

Long-running kernels accumulate drift that no restart-and-run-all discipline prevents during a session. The counter keeps climbing into the hundreds, cells get re-executed with edited source, and the counts in the file become a sparse sample of a long history. At that point the file is best read as evidence about a session rather than as a specification of a computation, and the specification has to be written down separately: an ordered script, a pinned environment, and a record of the inputs. The notebook can then remain what it is good at being, a readable narrative of the work, without carrying a claim it cannot support.