3.2 Build a scalar autograd engine
Autograd records the arithmetic of the forward pass as a graph, then walks it backwards once to deliver the gradient of the loss with respect to every number that went in.
01Computation is a graph of Values
Training a model means nudging every weight a little in the direction that lowers the loss (gradient descent, lesson 2.1). For that we need for every weight: how much the loss moves per unit nudge of that weight. An autograd engine computes all of them automatically from the code that computes . Here is how. Wrap every scalar in a Value object that stores four things: its number (data), the Values it was computed from (_prev), the operation that produced it (_op), and a slot for its gradient (grad, starting at 0). Running ordinary Python on such objects records the computation as a side effect. The expression below builds the graph in the viewport: 13 value nodes (the glass blocks) and 7 operation nodes (the spheres).
The emerald pulses are the forward pass: each op runs once its inputs exist, and its block lights up with its value. The viewport is at with , , , and target , which gives , , , and . The middle line of each block is data; the bottom line is grad, still 0.000 until the amber backward pass returns from the right.
Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd: the Value object
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.1
02Local derivatives and the chain rule
Each operation only needs its local derivative: how its output moves when one input moves and the others are held fixed. Four rules cover every op in this lesson:
Write for the gradient of the loss with respect to node , read v-bar.
The chain rule says that an op passes its own gradient back to each input , scaled by the local derivative:
- 1
The seed: the loss changes one-for-one with itself. backward() sets L.grad = 1 before anything else.
These are the numbers on the blocks once the amber pass has finished, and the bars on top show their size and sign: amber for positive, cyan for negative. Hover to read the local derivatives of the op the animation is on.
Freeze the pass where every gradient is ready and scrub it yourself.
Check the meaning of with a real nudge: move from 0.5 to 0.51 and goes from 0.16 to 0.1525, a change of . The gradient predicted . The small gap is curvature, which a straight-line prediction ignores.
The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018, Sec. 4.5, the chain rules
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.2
03Reverse mode: one backward pass, every gradient
A node may only pass its gradient back once every node that uses it has added its contribution. A topological order lists each node after all of its inputs, so walking it in reverse visits each node after all of its consumers. micrograd builds one with a depth-first search from that appends a node after all of its children. The small emerald number on each block is its position in that order: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. This is one valid order; yours may differ, because your Value stores _prev as a set and may visit inputs in another order.
Slow the animation to 0.3× with the indices on.
Each op does a constant amount of extra work in the backward pass, so one reverse sweep costs a small multiple of the forward pass and yields for every node at once. Forward mode differentiates the other way round: it carries for one chosen input alongside the values, so the full gradient needs one pass per input. That is 6 passes for the six leaves here, and passes for a network with parameters.
x₁, w₁, x₂, w₂, b, y. Reverse mode fills the whole vector in one backward pass; forward mode fills one entry per pass.
Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018, Sec. 3.1 and 3.2
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.3 and 6.5.9
The rest of this lesson is in Pro
- 04Fan-out: gradients accumulate
- 05A neuron: the tanh derivative
- 06From scalars to tensors: what PyTorch does
- 07Training on your engine
- and 1 more step
- Python challenge: Build a scalar autograd engine
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is autograd in PyTorch?
- Autograd is PyTorch's reverse-mode automatic differentiation engine. During the forward pass it records every operation on tensors that require gradients as a node in a graph. Calling backward on a scalar loss walks that graph in reverse topological order, applies each operation's local derivative with the chain rule, and adds the result into the .grad of every leaf tensor. The graph is rebuilt on every forward pass.
- Why does PyTorch accumulate gradients instead of overwriting them?
- A value used in several places affects the loss through several paths, and the chain rule adds those contributions, so the engine has to sum into .grad. The same mechanism lets you accumulate gradients over several micro-batches before one optimizer step. The cost is that you must reset gradients with optimizer.zero_grad() before each new backward pass, or every step uses the sum of all previous gradients.
- What is the difference between forward-mode and reverse-mode automatic differentiation?
- Forward mode carries the derivative with respect to one input alongside the values, so a full gradient needs one pass per input. Reverse mode runs the forward pass, then propagates the derivative of one output backwards to all inputs in a single sweep. Training has millions of inputs (parameters) and one scalar loss, so deep learning uses reverse mode, also called backpropagation; its price is storing the intermediate values of the forward pass.
Next in Deep Learning and Autograd
3.3Initialisation, normalisation and training stabilityWhy activations explode or vanish, and how Kaiming init, BatchNorm and LayerNorm keep signals well-conditioned.Sources
- Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy
- Deep Learning, Goodfellow, Bengio & Courville
- Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018
- The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018