Receptive Field Arithmetic and Effective Coverage

The receptive field of a unit is the set of input elements that can change its value. For a convolutional stack this set has a closed form, and the closed form is easy to compute wrongly, because stride, dilation, and pooling all enter the recurrence in different places. The arithmetic matters for a concrete reason: a segmentation or detection head emits one prediction per output cell, and if a target object is larger than the unit’s true field, no amount of training will let the unit see it whole.

The nominal field, however, is a rectangle that describes reachability, not influence. The part of the field that carries most of the gradient is markedly smaller and roughly elliptical. Distinguishing the two is what separates a layer that works on large objects from one that merely has a large parameter count.

The recurrence, term by term

Track two quantities per stage: the jump $j$, the distance in input pixels between adjacent output cells, and the field $r$, the width in input pixels spanned by one output cell’s ancestors. Both start at 1 before any layer is applied. For a convolution with kernel extent $K$, stride $S$, and dilation $D$, the effective extent is $K_{eff} = D(K-1) + 1$, and one stage updates

$$r \leftarrow r + (K_{eff} - 1), j, \qquad j \leftarrow j \cdot S.$$

The order is important: the field uses the jump as it was on entry, before the stride multiplies it. A 3×3 layer with stride 1 at the start gives $r = 1 + 2 = 3$, $j = 1$. A second such layer gives $r = 3 + 2 = 5$. A stride-2 3×3 layer after that gives $r = 5 + 2 \cdot 1 = 7$ and then $j = 2$: the field grows by the same 2, but every subsequent layer now adds twice as much.

From the third layer on, a 3×3 stride-1 convolution adds $2j$ to the field, so depth buys coverage at a rate proportional to the cumulative downsampling. This is why the deep stages of a classification backbone have fields of many hundreds of pixels while their kernels remain 3×3, and why the field of a network that has been truncated before its first stride reduction is much smaller than expected.

Stride appears in the field recurrence only through $j$; it never enters $K_{eff}$. That is a common source of confusion: a stride-2 layer with a 3×3 kernel does not span 5 inputs. It spans 3, at positions two apart in the output lattice, but consecutive output cells share overlaps such that the union over three output cells covers 7 inputs.

def receptive_field(layers):
    """layers: list of (kernel, stride, dilation) for one path through the net."""
    r, j = 1, 1
    for k, s, d in layers:
        k_eff = d * (k - 1) + 1
        r = r + (k_eff - 1) * j
        j = j * s
    return r, j

backbone = [(3, 1, 1)] * 3 + [(3, 2, 1), (3, 1, 1)] * 2   # schematic
print(receptive_field(backbone))          # field grows slower than depth suggests

Pooling obeys the same rule with $K$ as the window and $D = 1$, so a 2×2 stride-2 max pool is exactly a stride-2 convolution’s contribution to $r$ and $j$. A larger stride pool contributes proportionally more.

Effective field versus geometric field

The rectangle from the recurrence counts positions that reach the output through at least one path. The influence of a position is the sum over all paths, weighted by the product of kernel weights along each path. With random zero-mean weights the expected contribution of a position falls off with the number of path choices that reach it and with the distance from the centre.

For a stack of $L$ layers with kernel $K$, the number of paths from an input position at offset $u$ from the centre to the output is the coefficient of the corresponding term in the $L$-fold convolution of a box filter of width $K$, which is a discrete approximation to a Gaussian of standard deviation proportional to $\sqrt{L(K^2-1)/12}$. The practical consequence: at the corners of the nominal field, the number of paths collapses to one (the all-extreme-tap path), and that path is multiplied by $L$ weights each of which is one of the least-used taps. Empirically and analytically, the effective field is close to the central half of the nominal one.

Two cures are standard. Stacking more layers of small kernels produces a smoother path distribution than one wide kernel, because the path count is a binomial rather than uniform, a 7×7 kernel gives every offset inside the field at least one path with weight equal to a single tap, whereas three 3×3 layers give the offset at distance 3 only one path but the offset at distance 2 three paths. That smoother taper is a real, if secondary, argument for the 3×3 stack beyond the parameter count. The other cure is to accept that the network cannot see far, and to reduce the resolution instead: strided downsampling makes each output cell’s field cover a bigger slab of the input, at the cost of positional precision.

Dilation buys coverage at a fixed resolution

A dilated 3×3 kernel with $D=2$ has $K_{eff} = 5$ with the same 9 weights; $D=4$ gives 9. Because the added coverage comes without stride, the output lattice keeps its resolution, so the field grows while the positional granularity does not coarsen. Stacks of exponentially increasing dilation, 1, 2, 4, 8 in four layers, produce a field that grows exponentially in depth while every output cell remains on the original grid, the construction used for dense prediction.

The cost is the grid artefact. A single dilated layer samples the input at positions spaced $D$ apart, so between the taps lie positions the layer never reads. In a stack of layers with the same $D$ the gaps persist and the field is a lattice of isolated spokes rather than a connected region. Mixing dilation rates, or following a dilated block with a 1×1 convolution whose taps see all channels at a point, removes the worst of it; the fully connected version of that argument is made in dilated and separable convolution in segmentation.

The other artefact is at the border. Dilation with symmetric padding needs $P = D$ to preserve extent, so a $D=8$ layer must fabricate an 8-pixel border. With zero padding, that band is a large fraction of the input for small feature maps, and for a 32×32 map the pad touches a third of the rows. Dilation is therefore applied on the low-resolution stages of a network, where the map is small but the field must be large.

Measurement rather than derivation

The recurrence gives the bound; measurement gives the shape. A direct measurement perturbs one input element and records the change in a chosen output unit. With random weights the response is a broad, smooth hill whose extent as defined by a threshold is what matters in practice. With trained weights the response is more structured, since a filter tuned to a specific pattern has strong support only where that pattern can form, so effective fields measured on a trained model are typically smaller and less symmetric than those measured on the untrained one.

A cheaper measurement ascends the gradient: for a single output unit, take the gradient of its activation with respect to the input and inspect the norm across the spatial axes. This gives the sensitivity map including the sign, which distinguishes a filter that responds to a pattern (a positive lobe surrounded by a negative one) from one that responds to an isolated edge. The computation is one backward pass and no forward recomputation.

The numbers to expect, stated as bounds rather than results: the nominal field from the recurrence is a hard limit; anything the recurrence says is outside the field contributes exactly zero because no path exists. Anything inside contributes a nonnegative amount of at least the product of the smallest-magnitude weights along its one path, which for typical initialisations is extremely small but not zero. There is no threshold at which the field stops; there is only a decay.

Where the arithmetic breaks down

Three structural cases invalidate the simple recurrence. Skip and dense connections create shortcuts, so the field of a unit in a residual block is the union of the field through the shortcut and the field through the residual branch; the union can be much larger than either path alone, which is why residual networks see further than their depth suggests. Global pooling sets the field of every unit to the entire input at a single stage, collapsing the distinction. And attention or any nonlocal mixing operation connects positions at arbitrary distance in one step, making the field equal to the whole tensor regardless of the convolution stack that came before.

There is also the case of the input itself. A convolution on a zero-padded image sees the pad, and the pad is fabricated content that a unit near the border treats as real. For border units the field includes positions outside the image, so nominal and effective fields both exceed the input extent; the measurements above should be restricted to interior units, or the boundary rows excluded, or the numbers will be distorted by whatever the padding invents. The same reasoning applies to the circular padding used in cross-correlation and convolution in the forward pass, where the field wraps and the notion of “outside the image” disappears entirely.

For deployment the useful output of all this is a single check: for the smallest object the model must localise, the field at the prediction stage must be at least that object’s size in the input grid, and the effective field should be comfortably larger. The recurrence is the cheap way to compute the first condition; the gradient measurement is the cheap way to check the second.