3.4 Convolutions, receptive fields and feature maps
A convolution slides one small set of weights across an image, so every output cell asks the same local question, and stacking such layers widens the region each cell can see.
01Why convolution: local, shared weights
Start in one dimension. Take one row of a small image, 16 numbers, and a kernel of three weights. Put the kernel over three neighbouring pixels, multiply each pair and add: that is one output number. Slide one pixel to the right and do it again. The widget does this for row 12 of a handwritten 4 with the kernel , which computes right pixel minus left pixel.
The viewport shows the same digit as a 16 × 16 grid of bars; height and brightness are the pixel intensity of the , from 0 to 1. An amber 3 × 3 window slides across it. At every stop it multiplies the nine pixels under it by the nine kernel weights (listed on the card beside the window and tinted onto its taps), adds the products, and writes one cell of the feature map floating above: emerald bars for positive responses, rose bars for negative ones.
Three assumptions about images are built into that operation. Locality: a useful feature such as an edge depends on a small neighbourhood, so each output reads only a 3 × 3 patch. Weight sharing: an edge detector that works in one corner works everywhere, so every position reuses the same nine weights. Translation equivariance follows from the sharing: shift the digit one pixel to the right and the whole feature map shifts one cell to the right (away from the border).
A dense layer connects each of the 256 outputs to each of the 256 pixels. The convolution has 9 weights (plus one bias) whatever the image size. For a 224 × 224 RGB image mapped to 64 channels of the same size, dense needs about 4.8 × 10¹¹ weights; a 3 × 3 convolution needs 1,792 including biases.
The kernel in the viewport is the Sobel x filter. Watch the map fill: an emerald band runs along the left side of each stroke, where intensity rises from left to right, and a rose band along its right side, where it falls. Flat regions, black or white, give 0.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 9.1 and 9.2
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.1, from fully connected layers to convolutions
02One output cell: cross-correlation
In the simplest case, no padding and steps of one pixel, output cell is the window whose top-left corner sits on pixel :
The general form adds three knobs. Pad the input with rings of zeros to get ; move the window pixels at a time, so its corner is at ; and space the taps pixels apart:
- 1
Padding shifts the index by one: padded row 12 is image row 11. The widget below does this bookkeeping for any cell.
The HUD's row shows the cell being written and its value. The first row of the map runs slowly so you can follow the window; the right edge of the same stroke gives .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 9.1, the convolution operation
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.2, convolutions for images
03Output shape: stride, padding and dilation
Three hyperparameters set how many window positions fit. Stride is the step between positions. Padding adds zeros on every side, so border pixels are visited as often as interior ones. Dilation spreads the taps pixels apart, so the kernel spans pixels with the same weights.
- 1
The padded input the window moves over.
The viewport lays every map out in input coordinates: a cell is drawn at the centre of the patch it summarises (the whole layer lifted and set back), so with the 8 × 8 map has cells two pixels apart, and its last cell is centred on pixel 14, not 15.
Spread the taps: dilation with stride 1 and padding 1.
Same dilation, padding raised to 2.
Stride 3 with no padding.
PyTorch's channels-first layout. A 3 × 3 convolution with stride 2 and padding 1 maps it to 32 × 64 × 112 × 112.
One k × k × C_in filter per output channel, 1,728 weights plus 64 biases. Each output channel sums over all input channels; the lab's single-channel case is C_in = C_out = 1.
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.3, padding and stride
Hands-On Machine Learning, Aurélien Géron, Ch. 14, convolutional layers
The rest of this lesson is in Pro
- 04Classic kernels and learned filters
- 05Stacking: the receptive field grows
- 06Modern practice: 1 × 1, depthwise, residual, ViT
- Python challenge: Write conv2d, its output size and its receptive field
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- How do you calculate the output size of a convolutional layer?
- For input size n, kernel size k, stride s, padding p on each side and dilation d, the output size per axis is floor((n + 2p - d(k - 1) - 1) / s) + 1. For example, a 224 pixel input with a 3 by 3 kernel, stride 2 and padding 1 gives floor(223 / 2) + 1 = 112. With stride 1 and no dilation, padding (k - 1) / 2 keeps the size unchanged. The same formula applies to pooling layers.
- What is the receptive field of a CNN?
- The receptive field of a unit is the region of the input image that can influence its value. A single 3 by 3 layer sees 3 by 3 pixels; each further layer adds (k - 1) times its dilation times the product of all earlier strides, so four stacked 3 by 3 layers with stride 1 see 9 by 9 pixels. Strides, pooling and dilation make it grow faster. In practice the pixels near the centre of the field have much more influence than those at its edge.
- Is convolution in deep learning actually cross-correlation?
- Yes. Mathematical convolution flips the kernel before sliding it, while the Conv2d layers in PyTorch, TensorFlow and other libraries slide it unflipped, which is cross-correlation. Because the kernel weights are learned, the difference does not matter for training: the network would learn the flipped weights. It matters when you use a hand-designed asymmetric kernel such as an edge detector, where a flip reverses the sign of the response.
Next in Deep Learning and Autograd
3.5Recurrent networks, LSTMs and backprop through timeUnroll a recurrent cell through time, watch gradients vanish down the chain, and see how gating keeps a memory alive.Sources
- Deep Learning, Goodfellow, Bengio & Courville
- Hands-On Machine Learning, Aurélien Géron
- Maths, CS & AI Compendium, Henry Ndubuaku
- Dive into Deep Learning, Zhang, Lipton, Li & Smola