Skip to content
LearnMLByDoing
Free

0.4 Derivatives: slopes, tangents and the chain rule

30 minFree lessonbeginner3B1BKhanMMLDLKarpathy

A derivative is the steepness of a curve at one exact spot, found by shrinking a gap until a slanted line between two points becomes the tangent.

01Steepness between two points

Walk along the cyan curve f(x)=x2f(x)=x^2 in the viewport. It is gentle near x=0x=0 and steep further right. To put a number on that steepness, pick two points on the curve and join them with a straight line, the amber secant. Its slope tells you how fast the curve climbs, on average, between the two points.

Worked example. The emerald point is P=(1, 1)P=(1,\,1) and the amber point is Q=(2, 4)Q=(2,\,4), because 12=11^2=1 and 22=42^2=4. Going from PP to QQ you move 1 across (the run) and 4−1=34-1=3 up (the rise). Slope is rise divided by run: 3÷1=33\div1=3. The HUD shows the same number as secant slope.

msec=riserun=f(x0+h)−f(x0)h\tok{secant}{m_{\text{sec}}}=\frac{\text{rise}}{\text{run}}=\frac{f(\tok{x0}{x_0}+\tok{h}{h})-f(\tok{x0}{x_0})}{\tok{h}{h}}
Rise is the change in height between Q and P; run is the gap h between their x values.

Halve the gap: set h=0.5h=0.5, so QQ moves to x=1.5x=1.5.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Calculus 1: secant lines and average rate of change

02Shrink the gap: the secant becomes the tangent

The secant measures steepness between two points, but we want the steepness at PP alone. The trick is to slide QQ toward PP. As the gap hh shrinks, the amber secant swings round until it lies on top of the emerald tangent.

Worked example. The viewport now has h=0.1h=0.1, so Q=(1.1, 1.21)Q=(1.1,\,1.21). Rise =1.21−1=0.21=1.21-1=0.21, run =0.1=0.1, slope =2.1=2.1. With h=0.01h=0.01 the slope is 2.012.01, and with h=0.001h=0.001 it is 2.0012.001. The numbers close in on 22, the slope of the tangent at x=1x=1. Try it in the table: press Shrink h ÷ 10 and watch the emerald digits settle.

Loading figure
Each row divides hh by 10. Digits of the slope that already match the true slope turn emerald; the error column shrinks by about 10 each row. Switch on Use both sides to preview the trick the challenge uses.
f′(x0)=lim⁡h→0  f(x0+h)−f(x0)h\tok{derivative}{f'(x_0)}=\lim_{\tok{h}{h}\to0}\;\frac{f(\tok{x0}{x_0}+\tok{h}{h})-f(\tok{x0}{x_0})}{\tok{h}{h}}
The derivative at x₀: the value the secant slope settles on as the gap shrinks toward zero.

Press play on the whole story: hh shrinks from 11 to 0.0010.001, then the view zooms in on PP.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.1, Differentiation of univariate functions

03Zoom in until the curve looks straight

Here is another way to see the same idea. Any smooth curve, viewed under a strong enough magnifying glass, looks like a straight line. The slope of that line is the derivative. The viewport shows f(x)=x3−3xf(x)=x^3-3x with PP at x0=0.5x_0=0.5, where f(0.5)=0.125−1.5=−1.375f(0.5)=0.125-1.5=-1.375.

Zoom in 100 times around PP: set the zoom slider to z=2z=2.

Worked example. Because the curve is almost straight near PP, you can predict nearby heights with the tangent: start at the height of PP and add slope times step. For f(x)=x2f(x)=x^2 at x0=1x_0=1 (height 11, slope 22), a step of 0.10.1 predicts 1+2×0.1=1.21+2\times0.1=1.2. The true value is 1.12=1.211.1^2=1.21, very close. The smaller the step, the better the prediction.

f(x0+h)  ≈  f(x0)+f′(x0) hf(\tok{x0}{x_0}+\tok{h}{h})\;\approx\;f(\tok{x0}{x_0})+\tok{derivative}{f'(x_0)}\,\tok{h}{h}
The straight-line (linear) approximation: new height ≈ old height + slope × step.

Try a different curve: the sigmoid at x0=0x_0=0, zoomed about 32×32\times (z=1.5z=1.5).

04The derivative is a new function

So far we measured the slope at one point. Now let PP travel along the curve and record the slope at every stop. Plot each slope as a height in the lower panel and you get a second curve: the slope graph f′(x)f'(x). In the viewport PP sweeps along f(x)=sin⁡xf(x)=\sin x and the emerald trace below is drawn as it goes.

Worked example. At x=0x=0 the sine curve climbs at slope 11. At the top of the hill, x≈1.57x\approx1.57, it is flat: slope 00. At x≈3.14x\approx3.14 it falls at slope −1-1. Those three points, (0,1)(0,1), (1.57,0)(1.57,0) and (3.14,−1)(3.14,-1), lie on the emerald curve below, which is cos⁡x\cos x.

f(x)=sin⁡x⟹f′(x)=cos⁡xf(x)=\sin x\qquad\Longrightarrow\qquad \tok{fprime}{f'(x)}=\cos x
Read the lower curve as "how steep is the upper curve directly above me".

Pause at the top of the sine hill: x0=1.5708x_0=1.5708, which is π/2\pi/2.

Sweep the cubic f(x)=x3−3xf(x)=x^3-3x instead.

05Shortcut rules: powers, constants and sums

Shrinking hh works for any curve, but for common functions the limit has already been worked out once and for all. A handful of rules covers most of what you will meet. First, see why the slope of x2x^2 is 2x2x:

The slope of x² from the definition1/4
  1. 1
    (x+h)2−x2h\frac{(x+h)^2-x^2}{h}

    Rise over run for f(x) = x², at any x.

ddx xn=n xn−1ddx c f(x)=c f′(x)ddx(f(x)+g(x))=f′(x)+g′(x)ddx c=0\begin{gathered}\frac{d}{dx}\,x^n=n\,x^{n-1}\qquad \frac{d}{dx}\,c\,f(x)=c\,f'(x)\\[4pt] \frac{d}{dx}\big(f(x)+g(x)\big)=f'(x)+g'(x)\qquad \frac{d}{dx}\,c=0\end{gathered}
Power rule, constant factor, sum rule, constant. Here c is any fixed number.

Worked example. For f(x)=x3−3xf(x)=x^3-3x: the power rule turns x3x^3 into 3x23x^2; the term −3x-3x has slope −3-3; the sum rule adds them, so f′(x)=3x2−3f'(x)=3x^2-3. At x0=2x_0=2, where the viewport is, f′(2)=3×4−3=9f'(2)=3\times4-3=9. The HUD agrees: the secant with h=0.001h=0.001 has slope 9.00609.0060, and the true derivative is 9.00009.0000.

Loading figure
This starts on the same curve, x3−3xx^3-3x, with the point at x=1.5x=1.5: the rule gives 3×1.52−3=3.753\times1.5^2-3=3.75, and the measured slope agrees. Change the power, the coefficient and the linear term, and drag the point. Moving the constant dd lifts the whole curve but never changes a slope.

Find the bottom of the cubic's dip: move PP to x0=1x_0=1.

A negative slope: the parabola x2x^2 at x0=−1.5x_0=-1.5.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Calculus 1: power rule and basic differentiation rules

06The chain rule: nudges multiply

Most functions in ML are chains: one step feeds the next. Think of linked gears. Turning xx turns uu, and turning uu turns yy. The viewport shows three number lines stacked top to bottom. A small nudge of xx (cyan arrow) moves uu (amber arrow), which moves yy (emerald arrow).

Worked example. Take u=2xu=2x and y=u2y=u^2, at x=1x=1. Then u=2u=2 and y=4y=4. The inner gear has slope du/dx=2du/dx=2: uu moves twice as far as xx. The outer gear has slope dy/du=2u=4dy/du=2u=4 at u=2u=2. Nudge xx by 0.050.05: uu moves 0.10.1, and yy moves from 44 to 2.12=4.412.1^2=4.41, about 4×0.1=0.44\times0.1=0.4. Altogether yy moved about 2×4=82\times4=8 times as far as xx.

dydx=dydu×dudx=4×2=8\tok{chain}{\frac{dy}{dx}}=\tok{outer}{\frac{dy}{du}}\times\tok{inner}{\frac{du}{dx}}=4\times2=8
Chain rule: the slopes along a chain multiply. Check: y = (2x)² = 4x² has slope 8x, which is 8 at x = 1.
Loading figure
Turn the nudge or switch on the wiggle. The bars show the three nudges on one shared scale. The inner slope is linked to the viewport's aa in u=a⋅xu=a\cdot x.

Swap the outer function for y=sin⁡uy=\sin u, keeping u=2xu=2x and x=1x=1.

Build a one-input neuron: u=3xu=3x and y=σ(u)y=\sigma(u), at x=0x=0.

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd: the chain rule in backpropagation

Deep Learning, Goodfellow, Bengio & Courville, Sec. 6.5.2, Chain rule of calculus

Compiling scene
Loading editor

Questions learners ask

What is a derivative in simple terms?
A derivative is how steep a curve is at one exact point: how much the output changes for a tiny change in the input. You find it by drawing a straight line through two nearby points on the curve and moving them closer together; the slope of that line settles on the derivative. For y = x squared, the derivative at x = 1 is 2, so near that point the output grows about twice as fast as the input.
Why does machine learning need derivatives?
Training a model means adjusting its numbers to make its error smaller. The derivative of the error with respect to each number says which way to nudge it and how strongly the error will respond. Gradient descent repeats that nudge thousands of times, and backpropagation is the efficient way to compute all of those derivatives at once using the chain rule.
How do you calculate a derivative numerically in Python?
Evaluate the function a small step h on each side of the point and divide the difference by 2h: (f(x + h) - f(x - h)) / (2 * h), with h around 1e-5. This central difference is much more accurate than the one-sided version, and with NumPy the same line works on a whole array of x values at once. Libraries such as PyTorch compute exact derivatives automatically, and numerical derivatives are a common way to check them.

Next in Maths Primer

0.5Partial derivatives and gradients: walking downhillStand on a hilly surface, take the slope in each direction separately, and combine them into the gradient: the arrow that points straight uphill.

Sources

  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Deep Learning, Goodfellow, Bengio & Courville
  • Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy