03
Deep Learning and Autograd
Networks as stacks of space-folding functions, trained by an autograd engine you build yourself.
- Lessons
- 5
- Study time
- 3.1 h
- Access
- First lesson free
Syllabus
- 3.1
Backpropagation and non-linear decision boundaries
See a hidden layer fold two interlocked spirals apart, then hand-write the backward pass that teaches it to.
45 minFreeLive 3D lab and Python challengeintermediate- Explain why a linear model cannot separate interlocked spirals
- Visualise how a hidden layer with a non-linearity re-embeds the data
- Derive the backward pass of a 2-layer MLP with softmax cross-entropy
- 3.2
Build a scalar autograd engine
A micrograd-style Value class with topological-sort backprop — the core of every deep learning framework in 120 lines.
40 minProLive 3D lab and Python challengeintermediate- Represent computation as a DAG of differentiable operations
- Implement reverse-mode accumulation via topological ordering
- Train a tanh neuron on your own engine
- 3.3
Initialisation, normalisation and training stability
Why activations explode or vanish, and how Kaiming init, BatchNorm and LayerNorm keep signals well-conditioned.
32 minProLive 3D lab and Python challengeintermediate- Track activation variance through deep stacks
- Derive Xavier and Kaiming initialisation scales
- Compare BatchNorm and LayerNorm statistics
- 3.4
Convolutions, receptive fields and feature maps
Weight sharing as an inductive bias: kernels sliding over images, stacking into hierarchical receptive fields.
34 minProLive 3D lab and Python challengeintermediate- Compute output shapes for stride, padding and dilation
- Grow receptive fields through depth
- Compare hand-designed filters with what trained CNNs learn, and read feature maps
- 3.5
Recurrent networks, LSTMs and backprop through time
Unroll a recurrent cell through time, watch gradients vanish down the chain, and see how gating keeps a memory alive.
36 minProLive 3D lab and Python challengeintermediate- Unroll an RNN and compute its gradients with backpropagation through time
- Explain vanishing and exploding gradients from repeated Jacobian products
- Describe how LSTM and GRU gates create additive memory paths