Translational Weight Sharing and Feature Map Aliasing

A fully connected layer applied to a 256×256×3 image with 1024 hidden units holds $256 \times 256 \times 3 \times 1024 \approx 2 \times 10^8$ weights, and every one of them is specific to a pixel. Move the object 20 pixels to the left and the entire activation pattern changes. The same layer as a 3×3 convolution with 1024 output channels holds $3 \times 3 \times 3 \times 1024 = 27{,}648$ weights, learns a filter that is applied at all 254×254 valid positions, and produces a response that moves with the object. The parameter reduction is a factor of about 7,000; the behavioural change is that the layer now has a built-in assumption about how images work.

That assumption is the subject here: precisely what invariance a convolutional layer provides, what it fails to provide, and which architectural choices weaken or strengthen it.

Sharing defines an equivalence class over positions

Writing the layer at a single position makes the sharing explicit. For a kernel $w$ of shape $K_h \times K_w$, the response at position $(i, j)$ is $y_{i,j} = \sum_{m,n} x_{i+m,j+n} w_{m,n}$. The kernel does not depend on $(i, j)$. The alternative, a distinct kernel per position, is a one-to-one layer with $H W K_h K_w$ parameters. Weight sharing constrains all $HW$ kernels to be the same matrix, and that constraint is the entire inductive bias.

The consequence is equivariance rather than invariance, and the distinction is worth stating carefully because it decides what the rest of a network must do. Let $T_s$ denote translation by offset $s$. A correlation layer commutes with translation only up to the padding boundary: for an interior position,

$$y(T_s x) = T_s (y(x)),$$

so the feature map translates with the input. A max-pooling layer over a window of size $W_p$ then reports, per window, the maximum of its contents. If the object shifts by less than $W_p$, the set of values inside each pooling window may be unchanged as a multiset, so the pooled map can be identical; that is approximate invariance. Full invariance is never exact: a shift larger than the pooled window moves the peak to a different output cell, and a shift that crosses the image border changes the padded content.

The distinction has a practical form. A layer that is equivariant preserves positional information, which downstream layers need to compose features into larger structures: a corner detector’s response at the top left must not be indistinguishable from the same response at the bottom right, or the next layer cannot assemble them into an outline. A layer that is invariant discards that information. Convolutional stacks are equivariant all the way through and only become invariant near the classification head, by global pooling or by a flatten-and-dense step that has thrown position away. This ordering is deliberate and is the reason classification from a feature pyramid or a region proposal works at all.

import numpy as np

# circular correlation: no boundary effects, so equivariance is exact
def correlate_circular(x, w):
    kh, kw = w.shape
    h, wd = x.shape
    out = np.zeros((h, wd))
    for i in range(h):
        for j in range(wd):
            patch = x[(i + np.arange(kh)) % h][:, (j + np.arange(kw)) % wd]
            out[i, j] = (patch * w).sum()
    return out

x = np.random.default_rng(0).normal(size=(8, 8))
w = np.random.default_rng(1).normal(size=(3, 3))
s = 3
assert np.allclose(correlate_circular(np.roll(x, s, axis=1), w),
                   np.roll(correlate_circular(x, w), s, axis=1))

Under zero padding the assertion fails at the columns within one kernel radius of the border, because the shifted image brings different fabricated values into view. That is the first and least discussed violation of the guarantee: padding makes the operator position-dependent near the edges.

Stride and pooling coarsen the lattice

A stride-$S$ layer samples positions on a lattice of spacing $S$. The equivariance still holds for shifts that are multiples of $S$, because such a shift maps the sampled lattice onto itself. It fails for shifts in between: a shift of $S/2$ moves the sampled grid relative to the image, and the outputs differ. The response has become periodic with period $S$ rather than equivariant, and the magnitude of the discrepancy grows with the frequency content of the input relative to the sampling rate, the aliasing argument that motivates the antialiasing filters discussed in pooling operators as signal samplers.

Max pooling adds a second, nonlinear form of the same problem. Within a window, max is invariant to shifts that keep the maximum in place, and discontinuous when two near-equal values trade rank. The output is therefore invariant with respect to shift but not to content: a shift that moves a slightly larger value into the window changes the pooled output entirely. The failure is visible in the gradient, which routes entirely to the argmax and therefore jumps between input positions as the input shifts. Average pooling is linear and commutes with shift up to the boundary, so it is exactly equivariant for interior shifts and never discontinuous, at the cost of diluting peaks and being less selective.

The compound effect across a deep stack is that the network’s effective tolerance to translation is a product of the strides and pooling factors along the path. A total downsampling factor of 32, common in classification backbones, means the output grid has one cell per 32×32 block of input; shifting an object by a non-multiple of that block changes which cell is loudest, and hence can change the prediction. The approximate invariance at the logits comes from aggregation across many feature channels and from the spatial averaging at the head, not from any single layer.

Receptive field grows by composition

The field of input positions that can influence a given output unit follows a recurrence. Let $r_\ell$ be the receptive field size at layer $\ell$ and $j_\ell$ the cumulative stride. Then

$$r_\ell = r_{\ell-1} + (K_\ell - 1), j_{\ell-1}, \qquad j_\ell = j_{\ell-1} S_\ell,$$

with $r_0 = 1$ and $j_0 = 1$ for a single channel. A stack of three 3×3 stride-1 convolutions has $r = 1 + 2 + 2 + 2 = 7$, matching a single 7×7 kernel while using $3 \cdot 9 = 27$ weights instead of 49 and inserting two nonlinearities between the stages. That trade, more depth for fewer parameters and more nonlinearity at the same coverage, is what the VGG-era replacement of large kernels by 3×3 stacks established, and it is the origin of the now-standard “two 3×3 equals one 5×5” identity.

The recurrence also shows why a nominal receptive field overstates what a unit actually sees. For an isotropic random-weight network the influence of the receptive field’s centre on the output decays smoothly toward the edges; a Gaussian-like effective field replaces the rectangular bound. Positions in the corners of the theoretical field contribute with weight approaching zero, because a corner position requires every layer along the path to pick the extreme tap. Architectures that need a genuinely large field therefore add explicit mechanisms rather than relying on depth: dilated stacks, or the pyramid pooling used in the multiscale section of semantic segmentation and skip connections.

Stack	Params per channel pair	Coverage	Nonlinearities
one 7×7	49	7	1
three 3×3	27	7	3
one 5×5 dilated by 2	25	9	1

Weight sharing is not a claim about the world

The bias is a statement that a feature looks the same wherever it appears in the image frame. That holds for textures, edges, and material properties, and it fails for anything whose meaning depends on absolute position: the sky is at the top, faces are upright, a road vanishing point sits near the horizon. A network with exact weight sharing has no way to encode that. The standard remedies all reintroduce position-dependence in a controlled form. Positional channels concatenated to the input restore absolute coordinates to the first layer. Padding a border of a single distinguished value acts as a weak position signal because the fabricated boundary is detectable. The most common is simply the image border itself: the zero-padded frame is an absolute position marker that survives several layers, which is one reason a network trained on centred crops degrades on translated ones.

The other limitation is orientation. Sharing across spatial position says nothing about rotation or scale, so a filter tuned to a vertical edge responds weakly to the same edge rotated 45 degrees. Rotation equivariance requires either data augmentation, design symmetry (as in group-equivariant convolutions built from steerable filters), or pooling over transformed copies. None of these is free: the first costs samples and does not generalise to unseen angles, the second changes the operator and often the kernel’s structure, the third multiplies compute by the number of copies.

Parameters versus samples

The arithmetic of sharing is often stated as a parameter reduction, but the more consequential effect is on sample complexity. A dense layer must estimate a separate weight for every (position, feature) pair, so evidence that a feature is useful at one position gives no information about its usefulness elsewhere; the number of examples required grows with the number of positions. A shared kernel pools evidence across all $H_{out} W_{out}$ applications in every update, which is why convolutional models are trainable on datasets that are orders of magnitude smaller than their parameter count would otherwise permit.

The price for that statistical efficiency is that the assumption must be true. When it is violated, as in data where the same intensity value means different things in different regions, such as satellite bands indexed by absolute scene position, or spectrograms where the meaningful axis is frequency rather than time, the shared kernel underfits and the network spends capacity fighting the constraint. The architecture responds by mixing shared and unshared paths: the depthwise-then-pointwise, grouped, and attention-mixing variants in depthwise convolutions and spectral efficiency all relax the coupling in different directions, and the fully unshared limit of that family is a plain linear layer.