3.5 Recurrent networks, LSTMs and backprop through time
A recurrent network applies one cell again and again along a sequence, so its gradient is a long product of the same Jacobian: it fades or blows up like a matrix power unless a gate opens an additive path.
01Sequences need a state
Text, audio, sensor readings and clickstreams arrive in order, and their length changes from one example to the next. A feed-forward network takes a fixed-size input, so it cannot read a review of 12 words and one of 400 with the same weights unless you pad, truncate or average away the order. A recurrent network instead reads one element at a time and carries a summary forward: the state.
The simplest state is a running total. Reading the numbers 1, 2, 3 and keeping their sum gives : each step adds the new input to the old state, . Give the old state a weight and the total becomes leaky: with , gives , and an input steps old counts times as much.
The viewport shows the same idea with vectors. Each dark block is one copy of the cell, of them; the cyan bars under each copy are the input (three numbers per step) and the emerald bars above are the state (six numbers, drawn five times taller than their value). The green pulse runs left to right: a copy's bars appear only after the state has reached it.
Hands-On Machine Learning, Aurélien Géron, Ch. 15: Recurrent neurons and layers
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.1 Working with sequences
02The recurrence, unrolled
A vanilla RNN replaces the scalar weight with matrices and squashes the result with . In the row-vector convention of PyTorch and D2L, one step is
A batch of B sequences is processed in lockstep. The full input is a (T, B, D) tensor, read one time slice at a time.
Input-to-hidden weights.
Hidden-to-hidden weights, square because the state feeds back into itself. The parameter count, D·H + H·H + H, does not depend on T.
Hidden state after step t. Stacking all steps gives (T, B, H); a classifier usually reads only H_T, a tagger reads every H_t.
The scene's weights are chosen so the maths stays exact: with orthogonal, so every eigenvalue of has modulus , the cockpit's spectral radius (0.9 now), and so does every singular value. The inputs are a slow wave of amplitude 0.05, small enough that the units stay in the near-linear part of : the grey gauges above the copies show the mean slope per step, 0.996 on average here.
Raise the spectral radius to and watch the forward pass.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 10.1 Unfolding computational graphs; 10.2 Recurrent neural networks
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.4 Recurrent neural networks
03Backprop through time: a product of Jacobians
Let the loss read only the last state, with , so . Backpropagation through time (BPTT) is ordinary backprop on the unrolled graph: walk from step back to step 1, multiplying by the local Jacobian of each step. The rose ribbon is the result, its thickness at copy equal to .
- 1
One step back: multiply elementwise by tanh′, then by the transposed recurrent matrix. This is the line the challenge asks you to write.
The viewport is at , . The ribbon starts at on the right and thins to on the left, just under the bound : the HUD's factor per step is 0.895, times a mean a little below 1. This is lesson 1.1's picture: applying a matrix again and again scales each eigen-direction by its eigenvalue every time, so fades and grows. Hover to draw the bound as a grey outline on the ribbon.
Keep and unroll 60 steps.
Back to with .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 10.2.2 Computing the gradient in an RNN; 10.7 The challenge of long-term dependencies
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.7 Backpropagation through time
The rest of this lesson is in Pro
- 04Exploding gradients and clipping
- 05LSTM and GRU: gates and an additive path
- 06What the additive path buys: copy the first token
- 07Sequence models today
- Python challenge: Run, backpropagate and gate an RNN
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is backpropagation through time?
- It is ordinary backpropagation applied to a recurrent network after unrolling it: the network is drawn as one copy of the cell per time step, all copies sharing the same weights, and the gradient is passed backward from the last step to the first. Each step back multiplies by that step's Jacobian, and the gradient of the shared weights is the sum of the contributions from every copy. Truncated BPTT stops the backward pass after a fixed number of steps to bound memory and compute.
- Why do recurrent neural networks have vanishing gradients?
- The gradient that reaches an early time step is a product of one Jacobian per step, and each Jacobian contains the same recurrent weight matrix times the activation's slope. If the largest singular value of the recurrent matrix is below 1, and the tanh slope can only make it smaller, the product shrinks geometrically with distance, like a matrix power. After a few dozen steps the early inputs contribute almost nothing to the weight update, so the network cannot learn long-range dependencies.
- What is the difference between an LSTM and a GRU?
- Both add gates that create an additive path through time so gradients survive longer. The LSTM keeps a separate cell state with input, forget and output gates, four weight blocks in all. The GRU merges the cell and hidden state and uses an update gate that interpolates between the old state and a candidate plus a reset gate, three weight blocks, so it is smaller and often trains as well. Neither removes the need for gradient clipping.
Sources
- Hands-On Machine Learning, Aurélien Géron
- Deep Learning, Goodfellow, Bengio & Courville
- Dive into Deep Learning, Zhang, Lipton, Li & Smola