Pooling as a Downsampling Operator and Its Signal Effects

Pooling is the part of a convolutional network that changes resolution. Its stated purpose in early architectures was to buy tolerance to small input shifts and to reduce the spatial extent so that later layers cost less. Both claims are weaker than they appear: the tolerance is partial and content-dependent, and the cost reduction is achievable without discarding information at all. Reading pooling as a sampling operation with a nonlinearity attached makes clear what it does, what it destroys, and why the backward pass needs a rule for a maximum that has no derivative.

Two operators with different algebras

Average pooling over a window $W$ computes $\frac{1}{|W|}\sum_{(m,n) \in W} x_{i+m,j+n}$. It is linear: the pooled map of a sum of inputs is the sum of pooled maps, and the operator is a convolution with a box kernel followed by subsampling. The box kernel’s frequency response is a Dirichlet kernel, which is a lowpass filter with strong sidelobes and zeros at nonzero multiples of the sampling frequency: a fair antialias filter, not a good one.

Max pooling computes $\max_{(m,n) \in W} x_{i+m,j+n}$. It is a lattice operation: it commutes with monotone pointwise functions, so a ReLU before or after max pooling gives the same result, a commutation that makes the often-repeated advice to apply max pooling before the nonlinearity a matter of taste rather than arithmetic. It does not commute with the sum, so it is not linear, and it is not a filter in any useful sense: the pooled map of a weighted sum of two inputs is generally not a weighted sum of pooled maps.

The distinction matters for the gradient. Average pooling’s backward pass distributes the incoming gradient uniformly over the window: $\partial \bar{x} / \partial x_{i+m,j+n} = 1/|W|$. Max pooling’s backward pass routes the entire gradient to the position that held the maximum and zero to the rest, so a single pooled output value’s gradient reaches exactly one input. For a 2×2 window the gradient density differs by a factor of four per step, and over a stack of pooling layers the fraction of input positions receiving any gradient through a given path shrinks geometrically.

import numpy as np

def maxpool_backward(x, grad_out, size=2):
    h, w = x.shape
    ho, wo = h // size, w // size
    grad_x = np.zeros_like(x)
    for i in range(ho):
        for j in range(wo):
            win = x[i*size:(i+1)*size, j*size:(j+1)*size]
            k = np.unravel_index(np.argmax(win), win.shape)   # first max wins ties
            grad_x[i*size + k[0], j*size + k[1]] += grad_out[i, j]
    return grad_x

Tie handling is a real specification question rather than a detail. When two positions in a window hold the same value, argmax returns the first in index order, so the gradient goes entirely to one of them. An implementation that distributed the gradient across all tied maxima would produce a different update; frameworks differ in which convention they apply, and the difference is observable in the presence of exact ties, which is precisely what ReLU produces in abundance.

Overlapping windows and the pooling gradient

Overlapping pooling windows, meaning a stride smaller than the window size as in the 3×3 windows with stride 2 used in some early large-scale image models, make adjacent output cells share input positions. The forward pass is unchanged in character, but the backward pass accumulates gradients from both cells into the shared position, so the effective gradient magnitude at a shared position roughly doubles while the unshared positions keep a single route. The resulting parameter updates are uneven in a way that depends on the pooling geometry, which is why overlapping pooling was largely abandoned when stride-2 convolutions became the default downsampling mechanism.

The other convention question is what happens at the border when the extent is not divisible by the stride. Frameworks either drop the incomplete window, which loses input positions entirely, or pad with negative infinity for max pooling, which makes the pad never selected but changes the effective window shape at the boundary. For a 5×5 input with a 2×2 window and stride 2, three rows of output exist only if the fifth row is covered; with padding the covered window includes fabricated content, and the pooled value at the boundary is a max of real values because negative infinity can never win. With truncation the row is simply not represented anywhere downstream.

Stride is sampling, and sampling aliases

Max pooling with stride $S$ equals max pooling with stride 1 followed by taking every $S$-th element. The first part is a local nonlinearity; the second is decimation, and decimation without a lowpass filter folds high frequencies down into low ones. That folding is the source of the failure mode in which a small translation of an input by one pixel produces a large change downstream: the shift changes the phase of the high-frequency content relative to the sampling grid, and the folded energy moves.

Zhang’s analysis makes the point constructively: placing a lowpass filter before the subsampling step, a small blur with a stride-$S$-appropriate support, restores shift invariance that the plain operator lacks. The filter can be applied before or after the local maximum, since max commutes with monotone functions but not with blur; applying the blur first is what the sampling theorem asks for, and blurring after the max trades some of the locality of the maximum for smoother behaviour. Frameworks now often combine a stride-2 convolution with a deliberately chosen kernel shape rather than pooling at all, which achieves the same filtering effect inside the learned kernel.

The magnitude of the aliasing depends on the spectrum of the feature map. Early-layer features are dominated by high-frequency content, since edges are exactly the things that a box downsample destroys, so the aliasing is worst at the layers where the resolution is highest. Late-layer features are smoother by construction, and their downsampling is comparatively benign. This is an argument against uniform treatment of all stages, and it matches the empirical observation that the first stride reduction of a network is the most damaging to fine-grained tasks.

Global pooling is the terminal case

Global average pooling collapses the spatial axes to one value per channel. It is average pooling with a window equal to the whole map, so it destroys all spatial information and is exactly shift-invariant for interior shifts in the limit of a large map. Its appeal is parameter-free aggregation: the classification head becomes a single linear map from $C$ channels to the class count, and the network’s spatial inductive bias is retained until the final step rather than gradually erased. Replacing large dense heads with this operation is what allows a classification network to accept a range of input resolutions without a change in parameter count.

Global max pooling is the opposite choice, and it is strictly less informative in one direction: it responds to the strongest evidence anywhere in the map and is indifferent to how much of the map carries that evidence. An image with one small object and an image filled with the same texture produce the same global maximum. For fine-grained recognition the average is usually better behaved, because it counts. The mixed heads in modern networks take both, and the choice is a statement about whether presence or prevalence is the signal.

The structural limitation of both is that they cannot localise. A network ending in global pooling can say what is present but not where; the tasks that need both add a parallel decoder that keeps the spatial grid, which is the construction described in semantic segmentation and skip connections.

Learned and adaptive replacements

The obvious objection to pooling is that its aggregation function is fixed. Average pooling assumes every position in the window matters equally; max pooling assumes only one does. Neither is right for all features. Two broad replacements exist.

Learned pooling applies a small network to the window’s values, or, in the pooling-over-channels variant of squeeze-and-excitation, applies a global average and then a learned gate. The latter is not a replacement for spatial downsampling but an adaptive reweighting of channels, and it adds a small number of parameters and cost that scales with the batch and channel count rather than the spatial extent.

Soft pooling replaces the maximum with a smooth approximation, typically a weighted average with weights proportional to $e^{\beta x}$, which approaches the max as $\beta$ grows. This makes the operator differentiable everywhere, restores the distributed gradient that max pooling discards, and interpolates between average and max pooling as a hyperparameter. The cost is an exponential evaluation over each window.

Strided convolution subsumes the whole category: a stride-$S$ convolution with kernel extent $K$ learns its own aggregation over $K \times K$ inputs and subsamples, and it composes with the next layer’s channels. The reasons pooling survived in some designs are parameter count (zero), rigidity of the inductive bias (max cannot be talked into ignoring a peak by training), and the fact that a pooled maximum is invariant to contrast while a learned projection is not. The reasons it was removed elsewhere are the aliasing and the gradient sparsity described above.

Operator	Params	Gradient per window	Shift behaviour
average, 2×2/2	0	distributed, $1/4$ each	equivariant, then decimated
max, 2×2/2	0	all to argmax	invariant, with jumps
stride-2 conv, 3×3	$9 C_{in} C_{out}$	dense, learned	learned filter, then decimated

The learned alternative is usually an inverted-residual block built from the separable stages described in depthwise and grouped convolutions, whose strided depthwise stage performs the spatial filtering and the subsampling in one pass.

The counting argument for keeping stride

Every downsampling operation is a decision about where to spend compute. A network that never downsamples has the same cost per layer at every depth, so a 50-layer stack at full resolution costs 50 times a 2-layer stack. Stride and pooling reduce the spatial term by a factor of 4 per halving in each axis, which is why the canonical architecture spends most of its layers at low resolution and most of its parameters at high channel counts. Pooling is the cheapest way to get that reduction, with no parameters and no arithmetic beyond the comparison or the sum, and its cost is paid in information rather than in flops. That is the trade in one sentence, and the choice of which operator to use follows from which currency is scarcer in the deployment: parameters and arithmetic, or spatial accuracy.