Depthwise and Grouped Convolutions and the Cost per Channel

A standard convolution couples two things that could be separated: the spatial mixing of neighbouring values, and the linear mixing of channels at each position. Splitting them is the basis of the most consequential efficiency change in convolutional architecture after the small-kernel stack. A depthwise convolution applies one spatial filter per input channel and mixes nothing; a 1×1 convolution mixes channels and looks at one position. Together they compute a function whose parameter count and arithmetic are both a fraction of the dense kernel’s, and whose memory access pattern is so different that on some hardware the cheaper layer runs slower.

Grouped convolutions are the intermediate case: channels are partitioned into $g$ groups, each group is convolved independently, and the outputs of the groups are concatenated. The dense layer is $g=1$ and depthwise is $g=C$. Everything between is a dial, and the dial moves three quantities at once: parameters, arithmetic, and the amount of cross-channel information the layer can express.

The arithmetic of the split

For input channels $C_{in}$, output channels $C_{out}$, and a kernel of $K \times K$, the standard layer holds $C_{in}C_{out}K^2$ weights. A grouped layer with $g$ groups requires $C_{in}$ and $C_{out}$ to be divisible by $g$, and holds $C_{in}C_{out}K^2/g$: a factor of $g$ fewer. At $g = C_{in} = C_{out}$ the count is $CK^2$, and the spatial filters no longer interact across channels at all.

Adding the pointwise stage restores the cross-channel mixing. A depthwise separable layer has $CK^2 + C^2$ weights, against $C^2K^2$ for the dense layer with equal input and output widths. The ratio is

$$\frac{CK^2 + C^2}{C^2K^2} = \frac{1}{C} + \frac{1}{K^2}.$$

For $C = 256$ and $K=3$ that is about $0.115$: roughly a ninth of the parameters, and the same ratio in multiply-adds since each output element is one multiply-add per weight. The form of the expression is instructive. The $1/K^2$ term is the sharing of the pointwise stage across positions; the $1/C$ term is the collapse of the channel coupling. Width helps the first and hurts nothing; kernel size helps the second and is limited to the small values that keep the layer’s coverage at a stack of two or three.

The trade in expressiveness is not visible in this count. A depthwise separable layer cannot represent a filter that mixes channels and space in a coupled way: a kernel that responds to red-in-one-channel adjacent to green-in-another must be approximated by a spatial filter followed by a linear map, and the approximation is exact only for kernels that factor. That factorisation is the assumption being made, and it is the reason grouped variants are used in the stages where features are already largely separated by channel and avoided in the stages that must synthesise new joint features.

import numpy as np

def depthwise_separable(x, w_depth, w_point, stride=2, pad=1):
    c, h, w = x.shape
    kh, kw = w_depth.shape[1:]
    xp = np.pad(x, ((0, 0), (pad, pad), (pad, pad)))
    ho = (h + 2 * pad - kh) // stride + 1
    wo = (w + 2 * pad - kw) // stride + 1
    dw = np.empty((c, ho, wo))
    for ci in range(c):
        for i in range(ho):
            for j in range(wo):
                patch = xp[ci, i*stride:i*stride+kh, j*stride:j*stride+kw]
                dw[ci, i, j] = (patch * w_depth[ci]).sum()   # no channel sum
    return np.einsum('c hw, o c -> o hw', dw, w_point)       # 1x1 stage

The einsum line is the pointwise stage: a matrix product over channels at each position, with no spatial taps. The depthwise stage above it is the same sliding dot product as cross-correlation in the forward pass, restricted to one channel at a time, which means the loop nest has no reuse across channels in the inner loop, only across the batch.

Arithmetic intensity decides whether cheap is fast

The dense layer reuses each input value $C_{out}K^2$ times and each weight $HW$ times, so its ratio of flops to bytes is high and the hardware’s arithmetic units are the limit. The depthwise layer reuses each input value $K^2$ times (the neighbouring output positions of the same channel) and each weight $H_{out}W_{out}$ times, but it reads no other channel and writes no other channel from a given read. Per input byte it does about $2K^2$ flops; on hardware where the peak rate is measured in tens of flops per byte of bandwidth, the depthwise stage is bandwidth-bound and cannot reach peak arithmetic throughput no matter how the loops are arranged.

The pointwise stage is at the other extreme: it is a per-position matrix product, so each input value is read once per output channel tile and reused across the spatial extent, which is the dense, cache-friendly shape. Composing the two therefore gives a layer whose first half is bandwidth-bound and whose second half is compute-bound, with a transfer through memory between them. Fusing them into one kernel removes that transfer: the depthwise outputs are held in registers or shared memory for the tile of positions and consumed immediately by the pointwise product. Whether the fusion is possible depends on the tile size: a tile of $T$ positions needs $T \cdot C$ depthwise outputs live, which for larger $T$ or $C$ exceeds the register file.

The practical result is a layer that looks cheaper on a flop count and is not proportionally faster in wall-clock time. A depthwise separable block with one-ninth the flops may run at a third of the speed of the dense block it replaces on a device that was already saturating bandwidth, and faster than it on a device that was compute-limited. Any decision made from the parameter count alone is a guess.

Channel count versus device kind

This is why the efficiency literature branches. Mobile and embedded designs, whose arithmetic throughput is modest relative to bandwidth, take the full separable construction and reduce channel counts along with it. Datacenter designs, whose arithmetic is abundant, keep dense 3×3 convolutions for the early stages and use grouped variants mainly to reduce the channel-coupling cost in the heavy middle stages. The same operator appears in both with opposite justifications: on one side it saves bandwidth, on the other it saves arithmetic, and the ratio between those two costs differs by more than an order of magnitude across the two device classes.

The second-order effect is on the optimiser and the normalisation. A depthwise layer with $C$ filters has $C K^2$ parameters and receives a gradient proportional to the sum over positions of the input values in that channel. For a channel whose activations are mostly zero, as is common after a ReLU in a sparse stage, the gradient for the corresponding filter is small, and the channel can effectively freeze. Adding a normalisation step that keeps channel statistics centred before the depthwise stage reduces this; the usual placement of batch or layer normalisation between the two stages of a separable block follows from this and from the fact that the pointwise stage’s output scale is independent of the depthwise stage’s spatial averaging.

Grouping as a structural constraint

Grouped convolution was introduced to split a layer’s work across devices, and the group boundary was a synchronisation boundary: each group’s channels were computed independently and the results concatenated, so no communication occurred inside the layer. That origin explains the divisibility requirement and the awkwardness that follows: $C_{in}$ and $C_{out}$ must both be multiples of $g$, so the channel plan of a network that uses grouping is constrained at every stage.

The information-flow consequence is the more important one. Channels are partitioned arbitrarily and statically: the assignment of channel indices to groups is fixed at construction and enforced by the weight layout. A group of 32 channels can never influence another group within that layer; cross-group mixing must wait for a later pointwise stage or a shuffle operation. Grouping therefore bounds the layer’s ability to form features that combine information spread across the partition, and the usual remedy is to interleave channels between layers (a channel shuffle) so that the static partition changes between stages, or to include a pointwise stage that mixes across all groups.

Depthwise convolutions sit at the extreme: the partition is per channel, so no cross-channel interaction exists until the pointwise stage, and the factorisation assumption is maximal. The benefit is that the arithmetic is $1/C$ of the dense layer’s and the parameter count is $K^2$ per channel, which is the cheapest possible spatial operator that is still learned.

Rectification order and channel interaction

The placement of the nonlinearity between the two stages matters. A ReLU between depthwise and pointwise stages clamps the spatial response before mixing, so the mixed output is a linear combination of nonnegative spatial responses; the same ReLU after the pointwise stage clamps a linear combination of signed values. These are different function families, and the first restricts the pointwise stage to combining rectified spatial features. Moving the activation after the pointwise product restores the expressiveness of the dense layer’s ordering while keeping the parameter savings, at the cost of the normalisation statistics at the intermediate stage.

The other ordering constraint is stride. In a separable block the depthwise stage carries the stride, because moving it to the pointwise stage would subsample before the spatial filtering that produces the fine structure. A depthwise stage with stride 2 and a 3×3 kernel sees a $3\times3$ region per output at positions two apart, so adjacent outputs share only one row or column of input; the effective field of the block is therefore the depthwise coverage, not the product of the two stages. This is the same receptive-field accounting that governs any strided stack, described for the general case in receptive field arithmetic, and it means that replacing a dense strided layer with a separable one does not change coverage.

Where the split fails

Three failure modes recur. The first is representational: a task whose discriminative features are inherently cross-channel and local, such as a specific colour pair in a specific adjacency in some fine-grained and spectral classification problems, cannot be represented by a depthwise stage followed by a linear map unless the linear map happens to encode it, and the network spends capacity approximating a coupled kernel. The second is numerical: the reduction in channel coupling reduces the accumulated magnitude of each output, so the layer’s output variance falls relative to a dense layer with the same initialisation, and the initialisation has to be scaled accordingly or the early training signal weakens.

The third is structural and specific to depthwise layers: because each output channel depends on exactly one input channel, the layer cannot compensate for a degenerate input channel. A channel carrying only noise passes through to the output with the same noise, whereas a dense layer’s weights can learn to suppress it. In a network with many channels and a normalisation step per block, this is usually harmless. In a network with few channels or an untrained initialisation it shows up as an inability to fit the data at all, which is why depthwise stages appear in the middle of a stack and not as the first layer of a network operating on raw pixels.

The final boundary is the pointwise stage’s cost at large channel counts. The $C^2$ term dominates the $CK^2$ term once $C > K^2$, which for a 3×3 kernel means above about nine channels, so the separable block’s arithmetic is essentially the pointwise stage’s, and further savings must come from reducing width or from splitting the pointwise product too. The low-rank and grouped forms of the pointwise stage are the natural next step, and they reduce the same quadratic term at the same cost in channel coupling.