Cross-Correlation and Convolution in the Forward Pass

A convolutional layer takes an input tensor $x$ of shape $C_{in} \times H \times W$ and a kernel $w$ of shape $C_{in} \times K_h \times K_w$, and produces an output map. The operation performed is

$$y_{i,j} = \sum_{c}\sum_{m}\sum_{n} x_{c,,i+m,,j+n}; w_{c,m,n}.$$

Every index in the sum has the same sign on both operands. That is cross-correlation, not convolution. True convolution reverses the kernel before sliding it:

$$(x * w){i,j} = \sum{c}\sum_{m}\sum_{n} x_{c,,i-m,,j-n}; w_{c,m,n}.$$

The distinction is invisible in a single layer and irrelevant in a deep network, which is exactly why libraries take the shortcut and why the naming has confused readers ever since. What follows is the index arithmetic that both forms need, the conventions that determine output shapes, and the cases where the reversal stops being cosmetic.

Why both operations index the same kernel

The identity that connects them is $x * w = x \star \mathrm{flip}(w)$, where $\star$ denotes cross-correlation and $\mathrm{flip}$ reverses the kernel along both spatial axes. A convolutional layer that learns $w$ can represent either function, because the learned parameters are free. Training a network with cross-correlation and training the same network with convolution produce weights related by a flip, and the function computed by the network is identical. Since the flip is an involution and costs $K_h K_w$ swaps, it is omitted.

The omission matters in four places. A layer whose kernel weights are loaded from an external source, such as a signal-processing front end, a fixed edge detector, or a transfer of parameters from a paper that used true convolution, will compute the reversed filter unless the stored weights are flipped at load time. Libraries whose documentation describes the operation as convolution while implementing correlation produce off-by-flip errors in exactly these handoff situations. Second, the gradient with respect to the input of a correlation layer is a convolution with the same kernel, so an autodiff engine that implements only correlation must flip the kernel when propagating backwards. Third, the adjoint relationship that makes the operation self-dual in the symmetric-kernel case is lost: a symmetric kernel, such as an isotropic Gaussian or a Sobel operator that is symmetric about the axis it smooths along, gives the same result either way, which is why the bug is often missed. Fourth, the Fourier-domain route to a layer computes a true convolution by the convolution theorem and must conjugate one spectrum to match a spatial-domain correlation, a convention that a frequency-domain implementation has to state explicitly rather than inherit from the spatial one.

For an image, the consequence is quantitative rather than abstract. Flipping a 3×3 kernel whose weights are asymmetric turns a filter that responds to a bright-to-dark vertical edge into one that responds to the opposite transition. Both are legitimate features; the network will learn whichever it needs. The reversal only matters when a specific orientation is imposed rather than learned.

Output size is a bound on valid indices

Given input extent $H$, kernel extent $K$, padding $P$ applied symmetrically on each side, and stride $S$, the output extent is

$$H_{out} = \left\lfloor \frac{H + 2P - K}{S} \right\rfloor + 1.$$

The formula is not a convention to memorise but the count of starting positions at which the entire kernel still fits inside the zero-padded input. The first start index is $0$; the last valid start $t$ must satisfy $t + K - 1 \le H + 2P - 1$, so $t \le H + 2P - K$. The number of starts along a stride-$S$ grid is $\lfloor (H + 2P - K)/S \rfloor + 1$. When the numerator is negative the kernel cannot fit at all and the layer is invalid: a 5×5 kernel with stride 1 on an input narrower than five pixels and no padding produces no output rather than a partial one.

The same reasoning gives the condition for the output to exactly match the input extent, which is what a residual block needs if it adds a shortcut to its output. Setting the formula equal to $H$ requires $H + 2P - K = S(H-1)$, so $2P = (S-1)H + K - S$. For stride 1 that collapses to $P = (K-1)/2$, which is why odd kernel sizes are the default: a 3×3 kernel with $P=1$ preserves extent, a 5×5 with $P=2$ does the same, and an even kernel cannot preserve extent under symmetric padding at all, because $(K-1)/2$ is not an integer. Even kernels do appear in practice, including 4×4 kernels used with a one-pixel asymmetric placement in the stem of some classifiers, but the arithmetic has to be derived per layer rather than assumed.

def conv_out(size, k, s=1, p=0):
    num = size + 2 * p - k
    if num < 0:
        raise ValueError(f"kernel {k} does not fit in {size} with padding {p}")
    return num // s + 1

# extent preservation with stride 1: p = (k - 1) // 2 for odd k
print(conv_out(224, 7, s=2, p=3))   # 112
print(conv_out(224, 3, s=1, p=1))   # 224

Padding is a boundary condition, not a data value

Zero padding is a choice about what lies beyond the image, and it biases the operation near the border. A zero-padded region is indistinguishable from an image that is genuinely dark there, so a filter applied at the corner sees a fabricated edge between the image content and the pad. This is not a correctness bug, since the model adapts, but it does place a systematic gradient of activation magnitude across the spatial extent of every feature map, strongest at the edges and decaying inward by one kernel radius per layer. Deep stacks therefore see the effect compound: without padding, extent shrinks at every layer and information near the border is captured by progressively fewer units.

The alternative boundary conditions have different costs. Valid padding, $P=0$, removes the fabricated border entirely and shrinks the map by $K-1$ per layer, which is the historical choice for the LeNet-era stack of small images and is impossible for deep networks on modest inputs. Replication padding repeats the nearest pixel, removing the hard step to black; reflection padding mirrors across the boundary, making the first derivative continuous at the seam; circular padding wraps, which is appropriate for data with a periodic axis and wrong for images. None of these changes the output size, so they can be swapped per layer; all of them cost an extra pass over the padded buffer or a conditional in the inner loop. The mainstream choice is zero padding because it is expressible as a contiguous buffer with a known length, which makes vectorised gathers and the memory layout described in tensor strides and im2col expansion straightforward.

Stride and dilation reinterpret the index, not the kernel

Stride $S$ subsamples the set of valid starts. It is equivalent to running the layer at stride 1 and keeping every $S$-th output, so a stride-$S$ layer is not a different operation but a different sampling lattice. Dilation $D$ spaces the kernel taps apart: tap $m$ is applied at offset $mD$. The effective kernel extent becomes $D(K-1)+1$, and the output size formula replaces $K$ with that value:

$$H_{out} = \left\lfloor \frac{H + 2P - D(K-1) - 1}{S} \right\rfloor + 1.$$

Dilation multiplies receptive field size without multiplying parameters or arithmetic: a 3×3 kernel at $D=2$ covers 5×5 inputs with 9 weights. For extent preservation a dilated 3×3 layer needs $P = D$, which is why dilated stacks such as those used in dense semantic segmentation specify padding and dilation together, as the recurrence in receptive field arithmetic makes explicit. The subtlety is that dilation with stride greater than 1 can skip input positions permanently, the grid effect, because the sampled lattice on the input is the set of positions reachable as $iS + mD$; if $\gcd(S, D) > 1$, some residues are never touched and the output depends on only a fraction of the input. The failure is silent, since every output is still a valid sum.

Where the linear operation ends

The convolution itself is affine in the input and stores no state: the same kernel is applied at every position, which is the weight sharing that gives the layer its parameter count of $C_{in} K_h K_w$ regardless of image size. Everything that follows in a standard layer, including the per-channel bias, the normalisation across channels, and the pointwise nonlinearity, is separate arithmetic on the resulting map. Bias addition is worth noting because it is invariant across positions: it shifts the whole map, and if a subsequent batch-normalisation step with its own shift is used, the bias is redundant and is often dropped from the parameterisation, a detail that matters when loading weights between implementations that differ in whether the bias is stored.

The remaining boundary is numerical. The accumulation over $C_{in} K_h K_w$ terms in a single output value is a sum of products in the same order a dense layer would use, but the reuse pattern is different: each input value is read up to $K_h K_w$ times and each weight is read up to $H_{out} W_{out}$ times. Achieving anything close to peak arithmetic throughput requires the memory hierarchy to serve that reuse, which is the subject that the rest of this cluster builds on.