Residual Connections and the Identity Gradient Path

A residual block computes $y = x + F(x)$, where $F$ is an arbitrary sub-network of the same input and output width and the addition is elementwise. Removing the addition returns the block to a plain stack, whose gradient behaviour is the product of per-layer Jacobians described in saturating activations. Keeping it changes the algebra of that product in a way that is easy to state exactly.

The Jacobian of one block and of a stack

Differentiating the block gives

$$\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}$$

For a stack of $L$ such blocks, the Jacobian of the composition is the product $\prod_{k=1}^{L}\left(I + J_k\right)$ with $J_k = \partial F_k / \partial x_k$. Expanding the product gives the identity matrix plus every product of a subcollection of the $J_k$, in every order-preserving arrangement. The first term is the identity, obtained by choosing the identity from every factor; the remaining terms involve at least one $J_k$ and therefore represent paths that pass through at least one branch.

Two consequences follow directly from that expansion. First, the identity term does not depend on the magnitudes of the branch Jacobians, so no matter how small those are, the cotangent reaching an early block is not multiplied by a product of small factors alone. Second, the terms of order one in the branch Jacobians are sums of the individual $J_k$, so if the branch Jacobians are comparable in magnitude and uncorrelated in direction, those first-order terms add rather than multiply. The paper that introduced the construction describes the same structure in terms of paths through the network: of the $2^{L}$ paths from output to input, most pass through a few blocks, and all of them contribute.

What a bound on the branch buys, and what it does not

Suppose each branch Jacobian has spectral norm at most $\rho$. The norm of the product is at most $(1+\rho)^{L}$, which grows with depth, and the norm of the identity term alone is exactly 1. Neither of those numbers says much about the gradient in a specific direction, but the second is the relevant one when the branch Jacobians are small: a plain stack with per-layer factor $0.9$ attenuates a fifty-layer path by $0.9^{50} \approx 0.005$, while the same depth with residual addition has the identity path at exactly 1 and a first-order sum of fifty terms each of size at most $0.9$.

The property is a property of the addition, not a guarantee about the trained network, and the counterexample is easy to construct. If $J_k$ has an eigenvalue close to $-1$ in some direction, then $I + J_k$ is close to singular in that direction, and a product of such factors can attenuate the gradient there far more than a plain stack would. A branch whose output is trained to cancel its input occupies exactly that regime. The residual form shifts the initialization of the block Jacobians toward the identity and makes the identity path available; it does not prevent the parameters from moving away from it.

The variance of the activations is a separate matter with the same structure. If the branch outputs are roughly independent across blocks with variance $v$, the variance of the block output is $v + v$ rather than $v$ times a decay factor, so the forward signal does not vanish. What it does instead is grow linearly in depth until something else controls the scale. That is the reason the branch output is usually normalized or whose final weights are scaled before the addition; the addition itself contains no parameters to absorb the growth.

import numpy as np

def block_forward(x, W):                 # y = x + F(x), F = x @ W
    return x + x @ W

def block_backward(dY, x, W):
    dF = dY @ W.T                        # gradient into the branch
    dW = x.T @ dY                        # branch parameter gradient
    dX = dY + dF                         # identity term plus branch term
    return dX, dW

def plain_backward(dY, x, W):            # the same block without the addition
    return dY @ W.T, x.T @ dY

W = 0.5 * np.eye(3)
dY = np.ones((1, 3))
dX_res, _ = block_backward(dY, np.ones((1, 3)), W)
dX_plain, _ = plain_backward(dY, np.ones((1, 3)), W)
print(dX_res.max(), dX_plain.max())      # 1.5 vs 0.5

Where the branch is placed relative to the addition

The ordering inside the block determines whether the identity path is really the identity. If the addition is followed by a normalization and a nonlinearity before the value is passed to the next block, then the input to the next block is $g(x + F(x))$ rather than $x + F(x)$, and the Jacobian of the block acquires a factor $\partial g$ in front of the sum. With a rectifier, that factor is zero wherever the sum is negative, so the identity term is masked out on part of the input space, and with a normalization whose statistics are batch-dependent, the factor is not a function of the sample alone. The formulation that places normalization and nonlinearity inside the branch and ends with the bare addition restores the identity exactly, and networks following that convention have been trained to depths of a thousand layers and more on benchmark classification tasks, where the earlier ordering stops being trainable well before that.

Widths must match for the addition, which forces a choice when the block changes the number of channels. A parameter-free option does not exist; a projection with its own weight matrix $W_s$ used on the skip path gives $y = W_s x + F(x)$, whose Jacobian is $W_s + \partial F/\partial x$, replacing the identity with a learned linear map that can contract or expand. That is a strictly weaker guarantee than the identity, and it adds parameters and multiply-accumulates proportional to the projection’s size.

Memory and the alternative designs

The addition is elementwise and parameter-free: one pass over the elements, no multiply-accumulates, no additional storage for the parameters. Since it adds work proportional to the number of elements while adding no weights, it lowers the ratio of arithmetic to memory traffic in the block and pushes the block further toward the memory-bound side of the machine’s balance. Its main cost is in the backward pass, because the block input is needed to compute both the branch’s parameter gradient and its input gradient, and the identity path means the input tensor must remain alive until the reverse sweep reaches the block. A stack of $L$ blocks retains the inputs to all of them unless the forward pass is recomputed during the backward pass, so memory grows with depth at a rate set by the activation size and the batch size. This is a genuine trade against the alternative of not having the path at all.

Gated skip connections multiply the two paths instead of adding them, $y = T(x) \odot F(x) + (1 - T(x)) \odot x$ with a gate $T$ produced by the block. The gate has its own Jacobian, so the derivative of the skip path is no longer the identity times a constant but $\partial (1-T)/\partial x$ plus the $(1-T)$ factor, and the path can be closed by the gate. Concatenating earlier block outputs rather than adding them keeps every intermediate value explicitly, which grows the channel count and the memory of a deep stack linearly at best, and quadratically if every block reads all earlier outputs. The addition is the cheapest of the three in parameters and the most restrictive in what the training process can do to the path.

Failure modes that persist

Residual blocks do not fix diverging activations; growth linear in depth accumulates and only scale control elsewhere bounds it. They do not by themselves make a network’s function smooth: with rectified branches the composition remains piecewise linear, and the addition of two piecewise-linear terms increases the number of linear regions rather than reducing it, which is the accounting in linear region counting. They interact with stochastic masking, since dropout on the skip path removes the unconditional term for some samples, as covered under dropout masking. And they change the shape of the loss surface seen by the optimizer, since the identity path carries the input directly into the output; a residual network whose branches are initialized to produce near-zero output at the start behaves like a shallow network and grows its effective depth during training, which is why the initialization of the branch weights determines the depth the network can use early on.