Skip to content
LearnMLByDoing
Free

3.1 Backpropagation and non-linear decision boundaries

45 minFree lessonintermediateDLKarpathy

A hidden layer bends the input plane until two interlocked spirals can be split by a flat plane, and backpropagation is how the network learns that bend.

01Why a straight line cannot do it

The viewport shows two interlocked spirals: 150 emerald points (class 0) and 150 amber points (class 1), each arm winding about 1.1 turns around the origin. Every point is drawn at the height of its label, amber on top and emerald below. The translucent sheet is the model's current estimate of P(amber∣x)P(\text{amber} \mid \mathbf{x}), and training pulls it up to the amber points and down to the emerald ones. The cyan curve is where that estimate crosses 0.5: the decision boundary.

Start with the simplest classifier, logistic regression. It scores a point with one affine function and squashes the score into a probability. One point with friendly numbers: x=(0.5,−0.2)\mathbf{x} = (0.5, -0.2), weights v=(2,1)\mathbf{v} = (2, 1) and bias b=0b = 0 give the score 2⋅0.5+1⋅(−0.2)+0=0.82 \cdot 0.5 + 1 \cdot (-0.2) + 0 = 0.8, and sigmoid⁡(0.8)=1/(1+e−0.8)=0.69\operatorname{sigmoid}(0.8) = 1/(1 + e^{-0.8}) = 0.69: a 69% vote for amber. In general:

p(x)=sigmoid⁡(v⊤x+b),boundary: {x:v⊤x+b=0}\tok{probs}{p}(\mathbf{x}) = \operatorname{sigmoid}\big(\mathbf{v}^{\top}\mathbf{x} + b\big), \qquad \text{boundary: } \{\mathbf{x} : \mathbf{v}^{\top}\mathbf{x} + b = 0\}
Logistic regression: its decision boundary is always a straight line in the input plane.

The boundary is the zero set of an affine function, so it is a straight line. Whatever v\mathbf{v} and bb training finds, each side of that line contains whole arcs of both spirals, and accuracy stays far from 100%. More epochs or a better optimiser cannot fix this. The limit is the hypothesis class, not the search.

The viewport is running the network of this lesson with its activation switched off (Activation: none). It has 16 hidden units, but each one is a linear function of x\mathbf{x}, and a linear function of linear functions is still linear. So the whole model is one affine score, the cyan boundary is a straight line through the spirals, and the acc readout stalls at about 54% (chance is 50%).

Turn the non-linearity on: switch the activation to tanh and keep everything else.

Back to no activation, but with four times as many hidden units: H=64H = 64.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.1

02A hidden layer re-embeds the plane

A two-layer perceptron (MLP) first maps each input to HH new features, then runs logistic regression on those features. Start with one hidden unit. It is the same scorer as above with a bend at the end: a=σ(w1x1+w2x2+b)a = \sigma(w_1x_1 + w_2x_2 + b). The dashed line in the widget is where its score is zero; drag the weights and the input, then switch the activation.

Loading figure
One hidden unit is a line plus a bend: ReLU is zero on one side of its line and rises linearly on the other. The scene stacks 16 of them.

Stacking HH units side by side and feeding their outputs to logistic regression gives the full network. In the row-vector convention used by NumPy (each input x\mathbf{x} is a row):

h=σ(x W(1)+b(1))∈RH,z=h W(2)+b(2),p=softmax⁡(z)\tok{hidden}{\mathbf{h}} = \tok{sigma}{\sigma}\big(\mathbf{x}\,\tok{w1}{W^{(1)}} + \mathbf{b}^{(1)}\big) \in \mathbb{R}^{H}, \qquad \mathbf{z} = \tok{hidden}{\mathbf{h}}\,W^{(2)} + \mathbf{b}^{(2)}, \qquad \tok{probs}{\mathbf{p}} = \operatorname{softmax}(\mathbf{z})
Forward pass of the 2 to H to 2 network trained in the viewport.
X[,]600 values

one row per point

W1[,]32 values
h[,]4,800 values

the batch's hidden representation

Hidden unit jj computes σ(x⋅wj+bj)\sigma(\mathbf{x}\cdot\mathbf{w}_j + b_j), where wj\mathbf{w}_j is column jj of W(1)W^{(1)}. The affine part measures how far x\mathbf{x} lies past a line in the input plane; the non-linearity then bends that distance. A ReLU unit creases the plane along its line and flattens one side to zero, while a tanh unit turns it into a smooth step. Together, HH such features carry the flat input plane to a curved two-dimensional sheet inside RH\mathbb{R}^{H}.

Switch to ReLU and draw each hidden unit's line w1jx1+w2jx2+bj=0w_{1j}x_1 + w_{2j}x_2 + b_j = 0 on the input plane.

Drag Fold in the cockpit. Each point, the probability sheet and the deformation grid travel from input coordinates to a 3D view of h(x)\tok{hidden}{\mathbf{h}}(\mathbf{x}). Hidden space has 16 dimensions and a screen has three, so the view picks three directions. The vertical axis is the output layer's score direction, introduced in the next step. The two horizontal axes are the two directions in which the hidden activations vary most (a principal component analysis, lesson 2.7). Every axis is a linear function of h\mathbf{h}, so anything flat in hidden space stays flat in the view. With H=3H = 3 there is nothing to choose: the view shows all of hidden space.

See an uncompressed hidden space: three tanh units, fully folded.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.3

03In hidden space, a flat plane is enough

The output layer is logistic regression on h\mathbf{h}. With two classes, the softmax depends only on the difference of the two logits:

p(amber∣x)=ez1ez0+ez1=sigmoid⁡(w⊤h+c),w=W:,1(2)−W:,0(2),c=b1(2)−b0(2)\tok{probs}{p}(\text{amber} \mid \mathbf{x}) = \frac{e^{z_1}}{e^{z_0} + e^{z_1}} = \operatorname{sigmoid}\big(\tok{w2}{\mathbf{w}}^{\top}\tok{hidden}{\mathbf{h}} + c\big), \qquad \tok{w2}{\mathbf{w}} = W^{(2)}_{:,1} - W^{(2)}_{:,0}, \quad c = b^{(2)}_1 - b^{(2)}_0
Two-class softmax is a sigmoid of the logit difference.

Its decision boundary w⊤h+c=0\mathbf{w}^{\top}\mathbf{h} + c = 0 is a hyperplane in hidden space. The scene's vertical axis measures (w⊤h+c)/∥w∥(\mathbf{w}^{\top}\mathbf{h} + c)/\lVert\mathbf{w}\rVert, the signed distance to that hyperplane, stretched so the margin is easy to see. At fold 1 the hyperplane is the flat cyan plane at height 0, and every correctly classified point lies on its own colour's side: amber above, emerald below.

Now pull the fold back to 0 and follow the cyan contour. It is the same set, {x:w⊤h(x)+c=0}\{\mathbf{x} : \mathbf{w}^{\top}\mathbf{h}(\mathbf{x}) + c = 0\}, drawn in input coordinates. Because h\mathbf{h} is non-linear, a flat cut in RH\mathbb{R}^{H} pulls back to a curved boundary in R2\mathbb{R}^{2}. Training changes both: W(1)\tok{w1}{W^{(1)}} reshapes the sheet and w\tok{w2}{\mathbf{w}} tilts the plane.

Make the spirals noisy: raise the noise to 0.25, so the two arms overlap.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 15

04Backpropagation: the chain rule, run backwards

Gradient descent (lesson 2.1) moves every weight a little against its gradient ∂L/∂θ\partial \mathcal{L} / \partial \theta: how much the loss changes per unit nudge of that weight. The network in the viewport has 2H+H+2H+2=822H + H + 2H + 2 = 82 weights and biases at H=16H = 16, so it needs 82 gradients per update. Backpropagation gets all of them in one backward sweep. Before the matrix version, watch it on a network small enough to show every number.

Loading figure
Step the forward pass (values flow left to right), then the backward pass (gradients flow right to left). With the starting numbers p=0.589p = 0.589 for target y=1y = 1, so ∂L/∂s=p−y=−0.411\partial L/\partial s = p - y = -0.411, and ∂L/∂w11=(−0.411)(1.2)(0.712)(1)=−0.351\partial L/\partial w_{11} = (-0.411)(1.2)(0.712)(1) = -0.351: the product of the factors on the path from w11w_{11} to the loss. The cyan finite-difference readout agrees to 5 decimals. Click another weight to see its path.

Two things to notice in the widget. Every gradient is a product of local factors along the path from the weight to the loss, and the backward pass computes each shared factor once (here ∂L/∂s\partial L/\partial s and ∂L/∂zj\partial L/\partial z_j) and reuses it for every weight below it. When a number feeds two paths, as x1x_1 does, its gradient is the sum over the paths. With ReLU, a hidden unit whose zj<0z_j < 0 passes a local factor of exactly 0, so every weight feeding it gets zero gradient.

Why the output error is p − y (one example, two classes)1/3
  1. 1
    L=−[yln⁡p+(1−y)ln⁡(1−p)],p=sigmoid⁡(s)L = -\big[y \ln p + (1 - y) \ln (1 - p)\big], \qquad p = \operatorname{sigmoid}(s)

    Cross-entropy for a label y in {0, 1} and a predicted probability p of class 1, from the score s.

Training the full network minimises the mean cross-entropy over the NN training points:

L=−1N∑n=1Nlog⁡pn, yn\tok{loss}{\mathcal{L}} = -\frac{1}{N}\sum_{n=1}^{N} \log \tok{probs}{p}_{n,\,y_n}
Mean cross-entropy with integer labels y_n.

Backpropagation computes every gradient in one backward sweep, reusing the values stored during the forward pass. Stack the batch into matrices: XX is N×2N \times 2, h\mathbf{h} is N×HN \times H, PP holds the predicted probabilities and YY the one-hot labels, and A=XW(1)+b(1)A = XW^{(1)} + \mathbf{b}^{(1)} are the hidden pre-activations.

Backward pass of a 2-layer MLP1/5
  1. 1
    δ(2)=∂L∂Z=1N (P−Y)\tok{delta}{\boldsymbol{\delta}^{(2)}} = \frac{\partial \mathcal{L}}{\partial Z} = \frac{1}{N}\,(P - Y)

    The result derived above, for every row at once: predicted minus true distribution. The 1/N comes from averaging the loss over the batch.

delta2[,]600 values
delta1[,]4,800 values

same shape as h

Every gradient has the shape of the parameter it updates. Counting the work: the forward pass costs one matrix product per layer (XW(1)XW^{(1)}, then hW(2)\mathbf{h}W^{(2)}), and the backward pass costs two per layer, one for the weight gradient (h⊤δ(2)\mathbf{h}^{\top}\boldsymbol{\delta}^{(2)}) and one to send the error down (δ(2)W(2)⊤\boldsymbol{\delta}^{(2)}W^{(2)\top}). That is why a training step takes roughly three times the arithmetic of evaluating the network. Hover δ\tok{delta}{\boldsymbol{\delta}} to flash the points with the largest error ∣pn−yn∣|p_n - y_n|: they sit where the sheet has not yet reached their label, and they dominate the next update.

Loading figure
The loss and accuracy of the viewport's network, sampled ten times a second while it trains (loss on a log scale). Press Restart in the viewport to watch a fresh run.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd

05Capacity, dead units and gradient checks

The scene is now set to H=3H = 3 with ReLU, with the three units' lines drawn in amber. Three ReLU units crease the plane along only three lines, so the boundary is piecewise linear with a handful of pieces, while the spirals need many more. Accuracy plateaus near 70% however long it trains. Drag Fold to 1: with H=3H = 3 the folded view shows all of hidden space, rotated and stretched vertically, and nothing is projected away.

Give the network more creases: H=32H = 32 ReLU units.

ReLU adds a second failure mode. If unit jj has a negative pre-activation anja_{nj} for every input, then σ′(anj)=0\sigma'(a_{nj}) = 0 everywhere, its column of δ(1)\boldsymbol{\delta}^{(1)} is zero and its weights never change again. This is a dead unit: in the 2-2-1 widget above it is a hidden unit whose activation factor reads 0. With H=3H = 3, losing one unit removes a third of the model's capacity. Wider layers, smooth activations such as tanh and GELU, Leaky ReLU (a small slope instead of 0 for negative inputs) and careful initialisation make dead units rare.

Start every unit with a negative bias, b(1)=−1.5b^{(1)} = -1.5, so many of them begin switched off on the whole dataset (H=8H = 8, ReLU).

∂L∂θi≈L(θ+ϵ ei)−L(θ−ϵ ei)2ϵ,rel. error=∣gbp−gnum∣∣gbp∣+∣gnum∣\frac{\partial \tok{loss}{\mathcal{L}}}{\partial \theta_i} \approx \frac{\mathcal{L}(\theta + \epsilon\,\mathbf{e}_i) - \mathcal{L}(\theta - \epsilon\,\mathbf{e}_i)}{2\epsilon}, \qquad \text{rel. error} = \frac{|g_{\text{bp}} - g_{\text{num}}|}{|g_{\text{bp}}| + |g_{\text{num}}|}
Central-difference gradient check.

Before you trust a hand-written backward pass, compare it with central finite differences on a tiny network. The central difference has error O(ϵ2)O(\epsilon^2), so with ϵ=10−5\epsilon = 10^{-5} in float64 a relative error below 10−610^{-6} means the backward pass is correct; the challenge harness accepts anything below 10−510^{-5}. Errors near 10−110^{-1} usually mean a missing 1/N1/N, a missing transpose or the wrong activation derivative.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.3.1

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, makemore part 4: backprop ninja

Compiling scene
Loading editor

Questions learners ask

What does backpropagation actually compute?
Backpropagation computes the gradient of the loss with respect to every weight in one backward sweep through the network. It applies the chain rule layer by layer, starting from the output error p - y, and reuses the activations stored during the forward pass, so the cost is about the same as a forward pass.
Why does a neural network need a non-linear activation function?
Without a non-linearity, stacked linear layers collapse into a single linear map, so the decision boundary stays a straight line or hyperplane. Activations such as ReLU, tanh and GELU let hidden layers bend the input space so that classes which are tangled in the input, such as two spirals, become linearly separable in the hidden representation.
How do I check that my backward pass is correct?
Run a gradient check: for each parameter, compare the analytic gradient with the central difference (L(theta + eps) - L(theta - eps)) / (2 eps) on a tiny network, with eps around 1e-5 in float64. A relative error below 1e-6 means the backward pass is correct, and the challenge harness accepts anything below 1e-5; errors near 1e-1 usually point to a missing 1/N, a missing transpose or the wrong activation derivative.

Next in Deep Learning and Autograd

3.2Build a scalar autograd engineA micrograd-style Value class with topological-sort backprop — the core of every deep learning framework in 120 lines.

Sources

  • Deep Learning, Goodfellow, Bengio & Courville
  • Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy