Feature Definitions as Versioned Artifacts

A feature in a spreadsheet has one meaning. A feature in a system that trains models over months has several, and they diverge quietly. The column called user_orders_30d computed in March is not the same object as the column with that name computed in September, even if no one edited the query: the underlying source table gained a status column, a join started matching rows it previously rejected, or a time-zone default changed in the warehouse. Models trained on either version carry the coefficient learned from whichever one they saw, and nothing in the model file records which.

The remedy is not a better naming convention. It is to treat the definition of a feature as a first-class artifact with identity, provenance, and a version, in the same way that a compiled binary has a build number that can be traced back to source.

What a definition has to contain

A feature definition that is sufficient for a system to reproduce a training set contains five pieces, and each of them exists because of a specific way the feature can change underneath a model.

The computation. The query, or the stream job, expressed over named inputs. It must be parameterized by the point-in-time anchor rather than containing a literal date, or it cannot be replayed for a different window.

The input datasets, identified by snapshot. Not “the orders table” but the orders table as it existed on a particular date, or at a particular commit. A definition that names only the table can produce different results on the same anchor when the table’s contents are backfilled.

The entity key and timestamp semantics. Which column identifies the entity, and which column determines the time at which the feature value is valid. The second is the subtle one: an event timestamp and an ingestion timestamp differ by however long the pipeline lagged, and a feature defined on one cannot be evaluated against a model that expects the other.

The type and null policy. Whether the value is an integer count, a float, a string category, or a vector; and what the consumer should do with a missing value. A count that becomes null instead of zero after a join is a type change even though both serialize as numbers in a JSON payload.

The owner. A person or group accountable for the correctness of the value. The presence of a field in a schema does not constitute ownership; the artifact needs it explicitly, because the failure mode is a feature that nobody will change and nobody will check.

A feature store’s registry is usually exactly this set of fields, plus a physical location for the offline and online copies. The interesting design question is not the schema but the identity rule: what counts as the same feature across two definitions.

Content hashing versus sequential versions

Two conventions compete. The first assigns an integer or semantic version and increments it whenever the definition changes, with the change recorded in a changelog. The second derives the version from a hash of the definition’s canonical form, so that a definition with identical content always has the same identifier and any edit produces a new one.

Sequential versions are readable, which matters when a human is reading a model’s metadata at two in the morning. They also depend on the discipline of the person editing: a change that is applied directly to the physical table without touching the definition’s text does not bump the version and is invisible to the scheme. Content hashing inverts the trade. It cannot silently fail to notice an edit to the text, but it also assigns a new version when a comment changes, and it says nothing about whether two versions are semantically compatible.

Most registries combine the two: a human-assigned stable name, a content hash over the definition plus the hashes of its input snapshots, and a monotonic counter for ordering. The model references the hash, the display shows the name, and the ordering drives the “which is newer” question during backfill. The content hash has to be computed over a canonical serialization, sorted keys, normalized whitespace, explicit formatting of numeric literals, or two semantically identical definitions produce different hashes and the deduplication does nothing.

Change control on a moving dataset

The harder half of versioning is the data, not the query. A definition can be unchanged in every byte while producing a different value, because the table it reads has been backfilled, corrected, or repartitioned. Schema versioning alone does not catch this.

The practical answer is that a training run records, alongside the feature hashes, the snapshot identifiers of every input dataset it read. In a warehouse that supports time travel or snapshot isolation, that identifier is a table version or a change-sequence number. In a data lake it is a manifest list or a commit identifier from the table format. Where the storage provides no such notion, the only option is a copy of the data at training time, which is expensive but is what is being relied upon whether or not it is done.

With snapshots recorded, the reproducibility property becomes checkable: re-running the training job against the recorded feature hashes and snapshot identifiers must reproduce the same feature table, and therefore the same fitted model given the same hyperparameters and seed. The mechanisms that turn that check into a routine re-execution are covered in rebuilding a training run from its metadata. What matters here is that snapshot identity is part of the feature’s version, not an afterthought: a definition whose inputs are unpinned has no version at all in any useful sense.

Fan-in and the cost of a change

Versioning exists to make consequences visible, and the consequence that matters most is fan-in. A base feature such as an account’s tenure in days may be consumed by dozens of models and by other derived features. Changing its definition, say, to count from the first successful transaction rather than the first signup, which removes a population of never-activated accounts from the distribution, silently changes the input of every consumer.

Fan-in is characterizable statically: the registry knows which definitions reference which. That makes it possible to answer, before merging a change, which models would need retraining, and to require that the answer be recorded. The alternative is discovering fan-in through an incident, which is the same information arriving at a worse time. Registries that expose reverse dependencies as a query, “what reads this definition”, are worth more than registries that expose only forward lookups, because the forward direction is what the author already knows.

There is a matching discipline on the model side: a model’s metadata names the feature hashes it was fitted on, so a retrain that picks up a redefined feature changes those hashes automatically. The release check is then a comparison of two sets of hashes, which is mechanical, rather than a judgement about whether a diff to a SQL file looks significant, which is not.

The failure this prevents

Without versioned definitions, the failure is a model that cannot be explained. A metric degrades six months after deployment; the model was retrained twice; the feature table has been rebuilt by a pipeline migration; and the question “did the distribution of this input change” has no answer, because the old definition no longer exists and the old data no longer exists in the shape it had. Regression testing for models depends on being able to score a fixed evaluation set with a fixed feature computation and get the same number, and that requires both to be addressable artifacts.

The mechanics of pinning a model to a specific feature version in a serving path connect directly to the same problem in the other direction, where the concern is producing different features while the model stays fixed; that is the training-serving skew problem treated in the contract between features and models. Between the two, the invariant is simple to state and expensive to build: every number a model consumed during training must be identifiable by an artifact reference, and every number it consumes in service must be produced by a computation whose reference matches.