Feature Visualisation and What Convolutional Filters Reveal

A trained convolutional filter is a set of numbers with no visual meaning until something maps it back to the input space. The simplest interpretation reads the first layer directly: a 3×3×3 kernel is an RGB patch and can be displayed as one. Beyond the first layer that fails, because a deep unit’s input space is the output of every preceding layer, not pixels. Feature visualisation closes the gap by optimising an input image to maximise a chosen unit’s activation and displaying the result.

The method is simple to state and full of decisions. The objective needs several regularisers to produce anything interpretable, the optimised image is not unique, and the features it shows are better evidence about the model’s learned statistics than about the structure of the visual world.

The objective and its necessary terms

The base objective is a maximisation: find an input $x$ that raises activation $a_i^k(x)$ of unit $i$ in layer $k$. Optimising this alone produces an adversarial pattern: a high-frequency texture that drives the unit to extreme values through combinations of pixels no natural image would contain. The unit’s response is maximised; the picture is uninterpretable because the network’s sensitivity to pixel-level structure far exceeds its sensitivity to the shapes the unit was trained to detect.

Three regularisers are standard, and each suppresses a specific degeneracy.

An $L_2$ or total-variation penalty on the image removes the high-frequency content by making neighbouring pixels similar; the TV term penalises the sum of absolute differences between adjacent pixels and is the cheapest way to keep the optimised image within the frequency band that convolutions with small kernels respond to smoothly. A Gaussian blur applied after each optimisation step achieves the same effect by a different route and is easier to tune.

A perturbation-averaging term applies random small translations, rotations, and scalings to the image before evaluating the activation, then averages. Without it the optimisation finds one pixel pattern at one exact pose that maximises the response; with it the pattern must survive perturbation, and the surviving structure is the pose-invariant part, which is the part that resembles a feature rather than an artifact.

A contribution constraint limits each pixel’s influence, or clips the image to a range and normalises its distribution to look like the training data’s. The last is not cosmetic: a network trained on natural images has an input distribution with strong second-order structure, and an image outside that distribution reaches activations the model would never produce on real inputs.

import torch

def visualise(feat_fn, unit, steps=256, lr=0.05, tv_weight=1e-3):
    """feat_fn maps a batch of images to the chosen layer's feature map."""
    x = torch.randn(1, 3, 224, 224, requires_grad=True)
    for _ in range(steps):
        acts = feat_fn(x)
        loss = -acts[0, unit].mean()          # maximise the mean over positions
        tv = (x[:, :, 1:] - x[:, :, :-1]).abs().mean()
        tv = tv + (x[:, :, :, 1:] - x[:, :, :, :-1]).abs().mean()
        grad, = torch.autograd.grad(loss + tv_weight * tv, x)
        with torch.no_grad():
            x -= lr * grad                    # descending on the negated activation
            x.clamp_(-2.5, 2.5)               # stay near the input distribution
    return x.detach()

Two details in that loop change what the picture means. Averaging the activation over spatial positions rather than taking the maximum lets the optimisation find a pattern that repeats across the image, which favours textures; taking the maximum favours a single localised instance. And the regularisation weight is the parameter that most affects the result: too small and the image is noise, too large and it is a blurred blob that reveals the low-frequency bias of the layer rather than its selectivity.

What the first layer shows directly

The first-layer filters need no optimisation. Each is a small RGB patch, and after training on natural images they are oriented, bandpass, and localised, approximately Gabor functions with a preferred orientation, spatial frequency, and phase. The population covers orientation roughly uniformly, with more filters at low spatial frequency than a uniform sampling would give, and colour-opponent filters that respond to edges between hues rather than between luminances.

This outcome is the same as the sparse-coding result: a linear code learned on natural image patches to make the coefficient distribution as sparse as possible, with an overcomplete basis, recovers Gabor-like filters. The interpretation is that the first layer of a convolutional network is largely an efficient linear transform of its input, and the orientation selectivity is forced by the statistics of natural images rather than by the task. A network trained on random noise images produces first-layer filters that are not oriented, which is the control that establishes the dependence.

Across a trained stack, the progression is consistent: the early layers hold oriented edge and colour-opponent detectors, the middle layers hold texture and part-like patterns, and the late layers hold object-level configurations that are larger and sparser. The receptive field arithmetic in receptive field arithmetic explains why the size of the features must grow with depth; the visualisation adds that the kind of feature also changes, from local contrast to composite structure.

Maps, not single filters

A single filter’s visualisation shows what one unit prefers. The map, which is the response of one unit over all spatial positions, shows where the layer found that pattern in a given image. The two answer different questions and are often confused.

The map is computed by a forward pass, with no optimisation, and is therefore cheap and exact. For a deep layer the map has low resolution, so displaying it requires upsampling, and the upsampled map suggests precision it does not have: each cell covers a block of input positions whose size is the cumulative stride, and the boundary between two cells is not necessarily a boundary in the image. The saliency approach, the gradient of a chosen class score with respect to the input pixels, has the same property for a different reason: the gradient is defined at every input pixel, but its magnitude reflects the sensitivity of the score to that pixel rather than the contribution of the region containing it, and the two differ for any nonlinearity along the path.

Occlusion gives a more direct answer to “which regions does this prediction depend on”: slide a patch over the image, recompute the score, and display the drop. The result is coarse, since the patch size sets the resolution, and expensive, since it costs one forward pass per position. Its advantage is that it measures the effect of removing information rather than the local sensitivity, so it does not over-reward pixels whose change the network is locally steep to but globally insensitive.

The disagreement among these methods is the useful signal. A class score whose gradient is concentrated on the object but whose occlusion map is flat over the background has learned to use the object; the reverse indicates a reliance on context that the gradient method misses.

Why a visualisation is not a proof

The optimisation finds an input that maximises the unit. The space of such inputs is large, and the particular one that a gradient method starting from random noise reaches is determined by the optimiser’s path, the regularisers, and the random seed. Different seeds give recognisably different pictures for the same unit, and the shared structure across seeds is the part that can be attributed to the unit rather than to the search.

The second limitation is the distinction between selectivity and invariance. A unit that responds strongly to a visualised texture may also respond to many other inputs; a strong response to a synthetic maximum says nothing about the fraction of natural images that activate it. Measuring that fraction requires a dataset pass, not an optimisation, and the two numbers can disagree substantially.

The third is that the visualisation is a property of the model’s function, not of a concept. A network can implement a task with internal features that do not correspond to human categories, and the optimisation will still produce an interpretable-looking image if the unit happens to be aligned with something nameable. The ones that are not aligned produce the textures and repeated patterns that dominate published galleries; those are not artefacts of the method, they are units whose preferred input has no natural-image counterpart.

Adversarial direction and the same machinery

The optimisation that produces a feature visualisation is nearly identical to the one that produces an adversarial example: both ascend a gradient with respect to the input. The difference is the constraint. An adversarial example is constrained to remain close to a specific image in $L_\infty$ or $L_2$; a feature visualisation is constrained to lie near the distribution of the training data. The two studies meet at the same fact, that the network’s response is far more sensitive to input directions than the perceptible ones, and each is a statement about that fact from a different side.

This has a practical consequence for interpreting both. A saliency map that is faithful to the gradient is also faithful to the adversarial perturbation direction, so a region that the map highlights is a region where a small change would flow through to the output. That is a statement about sensitivity, and it is worth distinguishing from a statement about the model’s reasoning.

The limits of the layer-wise story

The progression from edges to textures to objects is a description of what is easiest to visualise, not a claim that the network partitions its computation by layer. A middle-layer unit can be sensitive to a large-scale pattern that the visualisation at its own resolution cannot show, because the pattern is assembled from features that the layer’s receptive field sees but the layer’s visualisation tends to render at a single scale. Varying the optimisation’s scale prior across runs reveals this: the same unit, optimised at different spatial scales, produces different structures, which means the unit has no single preferred scale.

The counter-evidence to a clean hierarchical reading is the observation that a deep convolutional network’s first-layer filters change little when the task changes, while the later layers change a great deal. The early layers are shaped by the input statistics; the later layers are shaped by the objective. The visualisation of a late layer therefore describes the task and the training data jointly, and transferring that picture to another model trained on different data is not justified.

What survives all of these qualifications is a small set of dependable claims. First-layer filters are oriented, bandpass, and colour-opponent, and this follows from natural image statistics rather than the training objective. Receptive fields grow with depth as the recurrence in translational weight sharing requires. Deep units respond to patterns assembled from the units below them, which is why a visualisation optimised for one unit and displayed at full resolution shows structure at the scale the unit’s field covers. And every visualisation is an optimisation with free parameters, so a picture without its constraints is not evidence about anything.