Stride-1 Backbones and Dense Prediction with Skip Connections

Dense prediction assigns a label to every input pixel. A classification backbone is built for the opposite contract: it reduces resolution by a cumulative factor of 16 or 32 and emits a single vector. Applying such a backbone to segmentation requires recovering the discarded grid, and the recovery is not interpolation, because no upsampling operator can reconstruct boundaries that were never in the coarse map. The architecture that solves it splits into an encoder that loses resolution but gains semantic depth, and a decoder that regains resolution by drawing on what the encoder produced at each level before discarding it.

The design question is therefore where the resolution and the semantics meet, and the several standard answers trade differently.

What downsampling discards cannot be interpolated back

At each stride reduction a spatial decision is made: which of the $S \times S$ input positions’ information survives into one output cell. A stride-2 convolution with a learned kernel keeps a weighted mixture of the $2 \times 2$ block; a max pool keeps the largest. Either way, the output cell no longer distinguishes the positions inside the block, so two pixels with different labels but the same block can only be separated by information from somewhere else.

Inside the receptive field, some of that information persists as adjacent-cell patterns, since a boundary within a 2×2 block perturbs the neighbouring cells too, but the perturbation is indirect and coarse. Beyond a few stride reductions the fine boundary structure is not recoverable from the deep map alone, which is why a segmentation model that predicts from the stride-32 map and upsamples bilinearly produces masks with boundaries quantised to the block size. The error is not random; it is aligned with the downsampling lattice, which is exactly the pattern that the operators in transposed convolution and the checkerboard artifact make more visible when used naively.

Skip connections solve this by keeping the pre-downsampling feature map and concatenating or adding it to the upsampled decoder map at the matching resolution. The decoder then has the semantic content from below and the boundary detail from the side. The two are at different scales, so the decoder must learn to reconcile them, typically with a convolution over the concatenated tensor that mixes both sets of channels.

The information-theoretic framing is precise. A skip connection from a stride-2 feature map carries $C \cdot H/2 \cdot W/2$ values, versus $C \cdot H \cdot W$ at full resolution; the full-resolution features exist only at the shallowest levels, where channels are few and semantics are weak. A U-shaped decoder with skips at every level reconstructs resolution using features from all levels, and the network can weight them by how much each contributes to each class boundary. That is the mechanism behind the encoder-decoder construction, and it costs memory: every skipped tensor must be kept alive until its decoder stage consumes it, so the peak memory includes the encoder activations at all stored levels at once.

def unet_shapes(h, w, widths=(64, 128, 256, 512)):
    """Encoder halves the extent; decoder doubles it and concatenates the skip."""
    enc = []
    for c in widths:
        enc.append((c, h, w))
        h, w = (h + 1) // 2, (w + 1) // 2
    dec, c, h, w = [], widths[-1], enc[-1][1], enc[-1][2]
    for sc, _, _ in reversed(enc[:-1]):
        h, w = 2 * h, 2 * w          # matches the extent of the skipped level
        c = c + sc                   # concat: upsampled channels plus skip channels
        dec.append((c, h, w))
        c = sc                       # the fuse convolution projects back down
    return enc, dec

enc, dec = unet_shapes(256, 256)
print(dec)   # (768, 64, 64), (384, 128, 128), (192, 256, 256)

Concatenation versus addition, and where the channels go

Concatenation gives the decoder both feature sets with separate weights, so it can learn a different linear response for each and cannot lose one to the other. The price is channel growth: the concatenated tensor’s width is the sum, and the following convolution’s parameter count is proportional to the product of the summed input width and the output width. Addition requires the two tensors to have the same channel count, which the encoder’s stage widths and the decoder’s stage widths must be designed to satisfy, and forces the two contributions into one shared weight matrix, where the network can only choose a compromise.

Residual backbones use addition so that the shortcut requires no parameters and the identity mapping is exactly representable. U-shaped decoders use concatenation because the skip carries information of a different kind, not a refinement of the same representation, and the extra parameters buy the separation. The choice is not stylistic: with addition the skip’s contribution is constrained to the decoder’s channel basis, and a boundary feature that lives in a channel no decoder stage uses has no path into the output.

Dilated backbones avoid the round trip

The alternative is never to lose the resolution. A dilated backbone keeps the highest-resolution stage at stride 1 or 2, replaces the stride reductions in the deep stages with dilated convolutions, and produces an output at the input resolution (or 4× reduced) directly, with no decoder. The field of the deepest unit still covers the whole image because dilation raises coverage without subsampling, as the recurrence in receptive field arithmetic shows.

The cost is paid in arithmetic. A stride-32 feature map is 1024 times smaller than the input, so a deep stage at stride 8 carries 16 times more positions than the same stage at stride 32. Each dilated stage therefore costs the flops of the full-resolution tensor convolved with a small kernel times the channel count, a substantial multiple of the encoder whose resolution it refuses to reduce. In practice dilated backbones apply the dilation only to the last block or two, accepting a stride-4 or stride-8 output and a small bilinear or learned upsampling to the end, and the designs that are fully dilated at stride 1 restrict the number of high-resolution channels.

The middle path, used in the encoder-decoder designs that dominate segmentation, is a backbone whose deepest stage uses dilated convolutions at stride 16 plus a decoder that brings the result from stride 16 to the input size with skips at the stride-4 and stride-8 levels. It avoids the stride-32 stage’s destructive loss of boundary information while keeping the deep semantic stage’s modest cost, and the number of skip levels is small enough that the stored activation memory is manageable.

Group normalisation and small batches

Dense prediction training is batch-limited. A full-resolution image with its mask must fit in device memory for training, so the batch size is often one or a few, and batch normalisation over a batch of one has per-channel statistics that estimate on a single sample. The statistics are still computed over the spatial extent, which for a large image is many samples per channel, so the estimate is not as degenerate as it is for a fully connected output; but the mismatch between the normalisation statistics at training and at inference, computed as running averages over small batches, grows.

Group normalisation removes the batch dimension from the statistics: it normalises over all channels within a group and over the spatial extent. For segmentation that makes the training behaviour independent of the batch size and consistent between training and inference, which is the reason the normalisation described alongside depthwise and grouped convolutions is often paired with grouped channel plans in a decoder. The cost is that the normalisation no longer has a cross-batch averaging effect, which for large batches is a regulariser and for small ones is noise.

The arithmetic consequence in the decoder is that the normalisation’s per-channel statistics must be computed over the spatial extent of every level separately, so a decoder with four levels does four reductions over tensors whose sizes decrease geometrically. The total cost is dominated by the topmost level, which is the full-resolution one.

Multi-scale aggregation at the end

Objects arrive at different scales, and a single-resolution feature map handles them unevenly: a small object occupies few cells at the stride-16 stage and may be below the resolution at which the stage’s filters can respond. Two cheap remedies are standard.

Spatial pyramid pooling applies pooling with several window sizes to the deepest map, then resamples each pooled result back to a common extent and concatenates. It gives every output position a summary of context at four or five scales, the same construction as the global pooling described in pooling as a downsampling operator, applied at several window sizes instead of the whole map. The parameter count is zero; the cost is the extra resampling passes.

Feature pyramid fusion goes the other way: it takes the decoder’s maps at each level, applies a top-down path that upsamples the coarser map and adds it to the finer one, and predicts from each level. The high-level semantics reach the fine levels without a second decoder, and the prediction is made at each scale on features that contain both. The addition requires channel projection at each level so that the widths agree, which is a small number of 1×1 convolutions.

Both approaches rest on the same principle: a prediction at a fine level benefits from context computed at a coarse level, and the transfer must be an explicit path rather than an accident of the receptive field. The check that it worked is a per-scale evaluation: a model whose accuracy falls steeply for small objects has the transfer at too coarse a level, and one that fails for large objects has the field too small at the fine levels.

Where the boundaries actually come from

The recurring lesson is that boundary quality is a function of how much full-resolution information the model still holds at prediction time, and of whether the loss distinguishes boundaries from interiors. A per-pixel cross-entropy over a class-imbalanced mask is dominated by the interiors of large objects, so the model can achieve a high pixel accuracy with blurred or shifted boundaries. Weighted losses, boundary-aware sampling, and the fine-level skips all address the same deficiency: the objective and the architecture must both carry resolution to the place where the label decision is made.

The residual failure mode is then structural rather than architectural, because an object smaller than roughly the finest prediction stride’s block cannot be localised more precisely than that block, and no amount of decoder depth recovers a distinction the lattice no longer holds. The remedy is a finer prediction lattice, which is the same dilation-versus-stride choice encountered above, and it is bounded below by the cost of computing a deep feature map at full resolution.