Skip to content
LearnMLByDoing
Free
03

Deep Learning and Autograd

Networks as stacks of space-folding functions, trained by an autograd engine you build yourself.

Lessons
5
Study time
3.1 h
Access
First lesson free

Syllabus

  1. 3.1

    Backpropagation and non-linear decision boundaries

    See a hidden layer fold two interlocked spirals apart, then hand-write the backward pass that teaches it to.

    45 minFreeLive 3D lab and Python challengeintermediate
    • Explain why a linear model cannot separate interlocked spirals
    • Visualise how a hidden layer with a non-linearity re-embeds the data
    • Derive the backward pass of a 2-layer MLP with softmax cross-entropy
  2. 3.2

    Build a scalar autograd engine

    A micrograd-style Value class with topological-sort backprop — the core of every deep learning framework in 120 lines.

    40 minProLive 3D lab and Python challengeintermediate
    • Represent computation as a DAG of differentiable operations
    • Implement reverse-mode accumulation via topological ordering
    • Train a tanh neuron on your own engine
  3. 3.3

    Initialisation, normalisation and training stability

    Why activations explode or vanish, and how Kaiming init, BatchNorm and LayerNorm keep signals well-conditioned.

    32 minProLive 3D lab and Python challengeintermediate
    • Track activation variance through deep stacks
    • Derive Xavier and Kaiming initialisation scales
    • Compare BatchNorm and LayerNorm statistics
  4. 3.4

    Convolutions, receptive fields and feature maps

    Weight sharing as an inductive bias: kernels sliding over images, stacking into hierarchical receptive fields.

    34 minProLive 3D lab and Python challengeintermediate
    • Compute output shapes for stride, padding and dilation
    • Grow receptive fields through depth
    • Compare hand-designed filters with what trained CNNs learn, and read feature maps
  5. 3.5

    Recurrent networks, LSTMs and backprop through time

    Unroll a recurrent cell through time, watch gradients vanish down the chain, and see how gating keeps a memory alive.

    36 minProLive 3D lab and Python challengeintermediate
    • Unroll an RNN and compute its gradients with backpropagation through time
    • Explain vanishing and exploding gradients from repeated Jacobian products
    • Describe how LSTM and GRU gates create additive memory paths