3.3 Initialisation, normalisation and training stability
Every layer multiplies the signal's variance by roughly n·Var(w), so a deep stack fades to zero, saturates or explodes unless initialisation and normalisation hold that factor near 1.
01Why deep stacks misbehave
The viewport holds a 16-layer MLP with 256 units per layer and no biases, evaluated on one fixed batch of 64 standardised rows. Each translucent curtain is the histogram of one layer's activations , with layer 1 in front. The wall at the left carries two series on a log scale: the emerald curve is each layer's standard deviation, and the amber bars show the backward signal. A wave carries the signal through the stack, then back.
Start with one number. Suppose each layer multiplies the size of its signal by the same factor . After one layer the signal is , after two , after sixteen . With that is ; with it is . Only keeps the signal the size it started. Drag below and above 1 and watch the decades pile up.
one per layer
one curtain in the viewport
- 1
Every pre-activation is a sum of n_in products.
The viewport uses the classic small init with tanh. Then , so each layer should shrink the std by . The measurement agrees: std , (a ratio of 0.158), and by layer 16 the std is : close to multiplied by itself sixteen times. From layer 2 on, every curtain is a needle at 0 and its colour fades to slate. The wall's log scale stops at , so the emerald curve runs into the floor of the wall, and a chip there gives the true value. Hover to draw the prediction beside the measured curve.
Backpropagation multiplies by the same weights, transposed, so it inherits the same factor. Near 0, and the backward factor is again 0.0256 per layer: the amber bars show . The first layer receives no usable gradient. This is the vanishing gradient problem, and its mirror image, factors above 1, is the exploding gradient problem.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2.5 and 8.4
Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, makemore part 3: activations and gradients
02Xavier initialisation for tanh
The fix is to choose so that the per-layer factor is 1. The forward pass sums over inputs and the backward pass sums over outputs, so the two directions ask for different variances. Glorot and Bengio split the difference.
- 1
Linearise around the origin, where a well-initialised tanh network operates.
In the viewport the curtains now keep a wide bell all the way down: std , , , and . The slow narrowing is real, not noise: , so tanh hands the next layer slightly less variance than it received, and the linear analysis is an upper bound. A fivefold change across 16 layers is benign compared with the of the previous step.
Use far too large weights: , so and pre-activations have a std near 16.
Sigmoid is worse. Its slope is at most 0.25, reached at . Even with Xavier weights, which keep , every layer multiplies the backward signal by at most 0.25, and fifteen layers give at most . Its outputs also sit around 0.5, not 0, which shifts every next layer's inputs. That is the textbook cause of vanishing gradients and why hidden layers moved to tanh, then ReLU.
Same Xavier weights, sigmoid activation.
Hands-On Machine Learning, Aurélien Géron, Ch. 11: Glorot and He initialization
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 5.4 Numerical stability and initialization
03Kaiming initialisation for ReLU
ReLU breaks the linear assumption in a precise way: it sets every negative pre-activation to 0. Xavier's variance then loses half of the signal at every layer. He et al. redo the bookkeeping with the second moment from the first derivation.
- 1
Zero-mean weights make every pre-activation symmetric, so half of them are clipped.
The viewport is set to Kaiming with ReLU. Each curtain is a slate post at 0, the exact zeros (about half of all entries), next to a half-bell, and the shape is the same at every depth: std , , and .
Keep ReLU but go back to Xavier's variance, instead of .
Kaiming sits exactly on the critical point, and the Weight gain slider in the cockpit lets you step off it. It multiplies every weight's std by a constant, so the per-layer factor becomes gain × 1. A gain of 0.9 already costs a factor of 5 in std over 16 layers (0.158 at layer 16), and a gain of 1.1 grows it by a similar factor (3.92): a 10% error in the init scale compounds just like the 0.16 of step 1, only slower.
Kaiming with ReLU, but every weight 10% too large.
A dead ReLU unit has slope 0 on every input, so no gradient ever reaches its weights and it cannot recover (lesson 3.1 shows one dying). The standard fixes change the slope on the negative side: Leaky ReLU uses there, and GELU and SiLU, the activations of most transformers, are smooth with a small non-zero slope for negative inputs. A small positive bias at init, a lower learning rate and normalisation also make dead units rarer.
Same Kaiming weights with Leaky ReLU.
Frameworks expose the choice as a mode. Kaiming's fan-in mode preserves forward variance; its fan-out mode, , preserves the backward signal instead. For square layers like these the two coincide.
Hands-On Machine Learning, Aurélien Géron, Ch. 11: Glorot and He initialization
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.4
The rest of this lesson is in Pro
- 04BatchNorm: normalise each feature over the batch
- 05LayerNorm, RMSNorm and where transformers put them
- 06Residuals, clipping, warm-up and diagnosing NaNs
- Python challenge: Initialise and normalise a deep stack
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between Xavier and He (Kaiming) initialization?
- Both draw zero-mean random weights and differ only in the variance. Xavier (Glorot) uses 2 divided by fan-in plus fan-out, which keeps the signal's variance roughly constant through tanh or linear layers in both the forward and backward pass. He (Kaiming) uses 2 divided by fan-in, twice as large, because ReLU sets half of its inputs to zero and would otherwise halve the variance at every layer. Use He for ReLU-family activations and Xavier for tanh, sigmoid or linear layers.
- Should I use BatchNorm or LayerNorm?
- BatchNorm normalises each feature over the batch, works well for convolutional networks trained with reasonably large batches, and needs running statistics at inference. LayerNorm normalises each example over its own features, so it does not depend on batch size, behaves the same in training and inference, and is the standard choice for transformers and recurrent networks. If your per-device batch is small, prefer LayerNorm or GroupNorm, or synchronise BatchNorm statistics across devices.
- Why does my training loss become NaN?
- Usually a value overflowed or an invalid operation occurred: exploding activations or gradients, a learning rate that is too high, the log of zero, or a float16 overflow above 65,504. Check the gradient norm and activation statistics just before the NaN. Typical fixes are a lower learning rate with warm-up, gradient clipping, proper initialisation and normalisation, numerically stable loss functions, and loss scaling for mixed precision.
Next in Deep Learning and Autograd
3.4Convolutions, receptive fields and feature mapsWeight sharing as an inductive bias: kernels sliding over images, stacking into hierarchical receptive fields.Sources
- Deep Learning, Goodfellow, Bengio & Courville
- Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy
- Dive into Deep Learning, Zhang, Lipton, Li & Smola
- Hands-On Machine Learning, Aurélien Géron
- Build a Large Language Model (From Scratch), Sebastian Raschka
- Attention Is All You Need, Vaswani et al., 2017