Linear Probing and Fine-Tuning as Different Measurements

The standard way to report a pretrained encoder’s quality is to attach a small classifier and measure accuracy on a downstream task. How much of the encoder is allowed to change during that measurement decides what the number means. Freezing the encoder and training only a linear layer measures the information that is already linearly separable in the features. Fine-tuning the whole model measures the best result reachable from that initialization. These are not two points on one scale; they are measurements of different properties, and a method that improves one can degrade the other.

The probe is a fixed readout, not a classifier

A linear probe fits a weight matrix W and bias b such that softmax(W·f(x) + b) predicts the label, with f frozen. The optimization is convex given f: a multinomial logistic regression with a unique optimum up to degeneracies. Whatever accuracy the probe reaches is a property of the feature space and the label, not of the probe’s training.

import torch
from torch import nn

def probe(features, labels, dim, classes, steps=1000, lr=0.1):
    clf = nn.Linear(dim, classes)
    opt = torch.optim.LBFGS(clf.parameters(), lr=lr)   # convex problem
    def closure():
        opt.zero_grad()
        loss = nn.functional.cross_entropy(clf(features), labels)
        loss.backward()
        return loss
    for _ in range(steps):
        opt.step(closure)
    return clf

The convexity is the point. Because the probe has no hidden layer and no nonlinearity, it cannot learn a new representation; it can only find a hyperplane that separates the frozen features. If the classes are not linearly separable in the feature space, no amount of probe training will help. The probe’s accuracy is therefore a lower bound on the information present, and a fairly tight one for the tasks where linear separability is the relevant question.

The probe’s hyperparameters still matter, and they are a source of cross-paper incomparability. Learning rate, weight decay, the number of steps, whether features are normalized, and whether the probe sees the same augmentation pipeline as pretraining all change the reported number by a margin comparable to the differences between methods. Feature normalization in particular can move accuracy substantially, because it removes the norm information that the encoder may have used.

Fine-tuning measures adaptability

Fine-tuning updates the encoder’s weights on the downstream task. It can reorient the whole feature space, undo invariances that the pretraining imposed, and specialize the representation to the new label set. Its result is an upper bound relative to the probe: with enough data and time, fine-tuning can reach the probe’s solution and improve on it, because the probe’s solution is in the fine-tuning search space.

The bound is not tight in practice. Fine-tuning on a small dataset with a high learning rate can destroy the pretrained features, ending below the probe. The phenomenon is well documented as catastrophic forgetting in the continual-learning literature, and the same mechanism applies: gradient steps that reduce the downstream loss on a narrow distribution move the weights away from the configuration that supports broad features. Lower learning rates, layer-wise decay, and freezing early layers are the standard mitigations, and each of them moves fine-tuning closer to the probe in behaviour.

Fine-tuning also changes what is being measured. A fine-tuned model that reaches high accuracy may have discarded most of the pretrained representation and relearned from the labelled data, so the number reflects the combination of pretraining and the fine-tuning recipe rather than the pretraining alone. When the downstream dataset is large, this is the expected outcome and the pretraining contributes mainly an initialization and a regularizer.

The gap carries the signal

Plotting probe accuracy against fine-tune accuracy for a set of pretrained encoders produces a scatter, not a line. Encoders with strong linear probes and weak fine-tuning exist, and so does the reverse. The four quadrants are informative.

High probe, high fine-tune: the features are linearly aligned with the label structure and remain adaptable. This is the usual target.

High probe, low fine-tune: the features encode the label but are brittle. The pretraining may have collapsed the space so that a single hyperplane separates classes while the underlying variation is gone, leaving nothing for fine-tuning to specialize. This pattern appears with objectives that optimize for cluster separation without preserving within-class structure.

Low probe, high fine-tune: the features are rich but not linearly organized. The information is present in a nonlinear arrangement that a linear readout cannot exploit, and fine-tuning can reshape the space to expose it. This is common with objectives trained for a different task, such as generative pretraining, where the features support the generative objective rather than discriminative separation.

Low probe, low fine-tune: the pretraining did not transfer, or the recipe is wrong.

The gap is also diagnostic of the augmentation policy. Invariance to colour, for example, reduces probe accuracy on tasks where colour is diagnostic, and fine-tuning can partially recover it by relearning colour sensitivity from the labelled data. The recovery is bounded by the information that survived the encoder, which is why the gap is larger for tasks that depend on the augmented factors.

Protocol details that change the ranking

Feature extraction depth. Using the output of the final layer is conventional but not obviously best. Intermediate layers often probe better for tasks that need spatial or local information, because the final layer has been compressed toward the pretraining objective. Reporting a single layer’s number hides this.

Pooling. A vision encoder that outputs a grid of features must be pooled to a vector for the probe. Mean pooling, max pooling, and using a designated class token give different results, and the choice interacts with the pretraining objective.

Augmentation at probe time. Evaluating the probe on clean images while the encoder was trained on heavily augmented ones measures a distribution shift that has nothing to do with the label. Evaluating with the same augmentation, averaged over views, measures something closer to the encoder’s actual capability.

Data volume. The probe’s advantage over fine-tuning shrinks as labelled data grows, because fine-tuning’s ability to reshape the space matters less when there is enough data to learn a good space from scratch. A comparison run at one data scale does not hold at another.

Choosing a protocol for a decision

If the question is whether the features are useful without further training, for example in a retrieval system where the encoder cannot be retrained per query, the probe is the right measurement and fine-tuning is irrelevant. If the question is whether the encoder is a good initialization, fine-tuning is the right measurement and the probe understates it. If the question is which of two encoders to ship in a system that will fine-tune, both numbers are needed, and the decision should weigh the fine-tuning cost against the probe accuracy that is available for free.

The probe is also the only honest measurement when the downstream consumer is not a trained classifier at all. A retrieval index, a clustering pass, or a duplicate detector reads the features directly and applies no learned transformation beyond a distance. For those consumers, the question is whether the geometry of the space already places related items together, and a probe over a labelled subset is the closest cheap proxy for that. Fine-tuning accuracy says nothing about them, because fine-tuning changes the space that the consumer would have used.

What the two numbers cost to obtain

The compute asymmetry is large enough to shape research practice. A probe can cache features once and then train a linear layer on them, so an epoch costs a matrix multiply; fine-tuning costs a full forward and backward through the encoder per epoch. This is why probe sweeps are cheap enough to run over many tasks and many encoders, and why fine-tuning sweeps are usually restricted to a few.

The probe also has a hyperparameter surface that is easy to search, because the problem is convex and a few hundred steps converge. Fine-tuning’s surface includes the learning rate, the schedule, the layer-wise decay, and the number of epochs, and each of these interacts with the others. Two papers comparing encoders by fine-tuning accuracy can disagree on the ranking purely because their fine-tuning recipes differ, which is not a criticism of either encoder.

# The two protocols differ only in what is frozen.
def run(f, head, x, freeze_encoder):
    if freeze_encoder:
        with torch.no_grad():
            z = f(x)              # features computed once, cached
        return head(z)
    return head(f(x))             # gradients flow into f

Caching is what makes the probe cheap enough to sweep. The features for a fixed encoder do not change, so one pass over the dataset produces a matrix that every probe configuration reuses, and the search reduces to fitting a linear model on a fixed matrix. Fine-tuning has no such shortcut, because every epoch changes the features and invalidates any cached matrix.

Reading a reported number

A reported probe accuracy is only interpretable alongside the details of the probe: the layer used, the pooling, the normalization, the number of labelled examples, and whether the images were augmented at evaluation. A reported fine-tuning accuracy is only interpretable alongside the schedule and the search budget. Papers in this area vary in how much of that they state, and the variation is large enough that cross-paper comparisons of absolute numbers are unreliable even when both numbers are correct.

The safer comparison is within one study, where the protocol is fixed and only the encoders change. That is the design of the large-scale adaptation benchmarks, and it is the reason their rankings carry more weight than the individual figures.

The invariance structure described in Data Augmentation as an Invariance Specification determines which of the two numbers a given encoder will do well on. A pipeline that removes colour makes the probe worse on colour-dependent labels while leaving fine-tuning free to relearn the sensitivity, so the gap widens for exactly the tasks where the augmentation was strongest. The spread of the space, discussed in Alignment and Uniformity as Separate Objectives, determines how much a linear readout can extract before fine-tuning has to reshape anything: a crowded space gives a probe little margin even when the information is present.

Two further measurements are worth reporting alongside the pair. One is the probe accuracy as a function of the number of labelled examples, which shows how quickly the representation becomes usable; the curve’s slope near the low-data end is often more decision-relevant than its plateau. The other is the accuracy of a probe trained on randomly initialized features of the same architecture, which is the floor against which any transfer claim should be read. That floor is not zero, because random features still separate many datasets reasonably, and the margin above it is the part attributable to pretraining.