Skip to content
LearnMLByDoing
Free

0.5 Partial derivatives and gradients: walking downhill

30 minFree lessonbeginner3B1BMMLKhanMLCC

A function of two inputs is a landscape, and the gradient is the arrow that tells you which way is uphill, so walking against it, one small step at a time, is how a model learns.

01A function of two inputs is a landscape

In Lesson 0.1 a function took one number and gave back one number, and its graph was a curve. Now the function takes two numbers, xx and yy, and still gives back one. Picture the floor of the viewport as a map: every spot on it has an xx and a yy, and above each spot the ground is raised to the function's output. The result is a landscape. The one on screen is a round bowl, and the white dot is the point x=1x = 1, y=2y = 2.

Worked example. The bowl's rule is: square each input, then add. At the white dot that is 12+22=1+4=51^2 + 2^2 = 1 + 4 = 5, so the ground there is 5 units high, and the HUD reads f(x,y)=5.000f(x, y) = 5.000. At the centre, 02+02=00^2 + 0^2 = 0: the bottom of the bowl, marked with a small emerald ring.

f(x,y)=x2+y2,f(1,2)=12+22=5\tok{height}{f}(x, y) = x^2 + y^2, \qquad \tok{height}{f}(1, 2) = 1^2 + 2^2 = 5
Two inputs in, one height out.

The faint rings on the floor are contour lines, like on a hiking map: each one joins points of equal height. The white dashed loop is the contour through your point, so every point on it has height 5. The small white walker follows it round and round, and the HUD's walker readout stays at f=5.000f = 5.000 the whole way. Notice that the rings crowd together further out: crowded contours mean steep ground.

Switch to the two hills landscape and put the probe at x=1x = 1, y=1y = 1, on the side of a hill.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.2

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: thinking about multivariable functions

02Partial derivatives: slice the landscape and look at one curve

Suppose you are only allowed to walk east and west, along xx, never north or south. The ground under your feet then traces a single curve. In the viewport the cyan sheet slices the bowl along the line y=2y = 2, and the cyan curve is that slice. The slope of that curve at your point is called the partial derivative of ff with respect to xx. The amber sheet does the same thing in the other direction: it slices along x=1x = 1, for walking along yy only.

Worked example. Freeze y=2y = 2. The bowl becomes f(x,2)=x2+4f(x, 2) = x^2 + 4, an ordinary one-input curve. From Lesson 0.4, the slope of x2x^2 is 2x2x, and the +4+4 is a constant that does not change the slope, so at x=1x = 1 the slope is 22. Now freeze x=1x = 1 instead: f(1,y)=1+y2f(1, y) = 1 + y^2, with slope 2y2y, which is 44 at y=2y = 2. The HUD agrees: ∂f/∂x=2.000\partial f/\partial x = 2.000 and ∂f/∂y=4.000\partial f/\partial y = 4.000. From this spot, walking along yy is twice as steep as walking along xx.

∂f∂x=2x=2,∂f∂y=2y=4at (1,2)\tok{partial_x}{\frac{\partial f}{\partial x}} = 2x = 2, \qquad \tok{partial_y}{\frac{\partial f}{\partial y}} = 2y = 4 \qquad \text{at } (1, 2)
Each partial slope treats the other input as a fixed number.
Loading figure
The slice f(x,2)=x2+4f(x, 2) = x^2 + 4. Nudge xx from 11 to 1.51.5 (h=0.5h = 0.5): the rise is 6.25−5=1.256.25 - 5 = 1.25, and rise over run is 1.25/0.5=2.51.25 / 0.5 = 2.5. Shrink the nudge to h=0.1h = 0.1 and you get 0.21/0.1=2.10.21 / 0.1 = 2.1; at h=0.01h = 0.01, 2.012.01. The estimates close in on the exact slope, 22.

Back in the viewport, the small cyan walker slides back and forth along the cyan slice carrying a short tangent line. Watch the tangent tilt as it moves: as the walker passes x=0x = 0, the bottom of the slice, the tangent goes flat and the walker readout's ∂f/∂x\partial f/\partial x changes sign.

Move the probe to the other side of the bowl: x=−1x = -1, y=2y = 2.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: partial derivatives

03The gradient: both slopes in one arrow

You now have two slopes at one point. Put them together as one arrow: go ∂f/∂x\partial f/\partial x units in the xx direction (the cyan leg), then ∂f/∂y\partial f/\partial y units in the yy direction (the amber leg). The emerald arrow from the start of the first leg to the end of the second is the gradient. It is the vector addition, tip to tail, from Lesson 0.2.

Worked example. At (1,2)(1, 2) the two slopes are 22 and 44, so the gradient is the arrow (2,4)(2, 4). Its length, by Pythagoras, is 22+42=20≈4.472\sqrt{2^2 + 4^2} = \sqrt{20} \approx 4.472: the HUD's ∣∇f∣|\nabla f|. In the viewport the three arrows lie flat against the bowl at your point, along the two tangent lines, and they are drawn at 0.2 times their true length so that they fit.

∇f(x,y)=(∂f∂x, ∂f∂y)=(2x, 2y),∇f(1,2)=(2, 4)\tok{gradient}{\nabla f}(x, y) = \left(\tok{partial_x}{\frac{\partial f}{\partial x}},\ \tok{partial_y}{\frac{\partial f}{\partial y}}\right) = (2x,\ 2y), \qquad \tok{gradient}{\nabla f}(1, 2) = (2,\ 4)
One partial slope per input, stacked into a vector.
∇f (gradient)[]2 values

One slope per input. A model with a million weights has a gradient with a million entries, one partial slope per weight, and the same rules apply.

Two facts about this arrow are visible in the viewport. First, it points uphill: away from the bottom of the bowl. Second, it crosses the white dashed contour at a right angle. The walker on the contour carries its own small emerald gradient arrow, and wherever the walker is, that arrow sticks straight out of the line. This makes sense: along a contour the height does not change, so all the climbing must happen across it.

Switch to the stretched bowl f=0.5x2+2y2f = 0.5x^2 + 2y^2 and move the probe to x=2x = 2, y=1y = 1.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.2

04Why the gradient is the steepest way up

Stand on a hillside and turn slowly on the spot. Facing one way the ground rises in front of you, facing the other way it falls, and in two directions in between it is level: those two run along the contour. The gradient points in the one direction where the ground rises fastest.

Worked example. The slope you feel walking in a direction uu (an arrow of length 1) is the dot product from Lesson 0.2: multiply matching entries and add. At (1,2)(1, 2) on the bowl, with ∇f=(2,4)\nabla f = (2, 4):

The slope in any direction1/4
  1. 1
    u=(1,0):2×1+4×0=2u = (1, 0):\quad 2 \times 1 + 4 \times 0 = 2

    Facing along +x you feel exactly ∂f/∂x.

Loading figure
Drag the white point (the 3D probe follows it) and swing the direction dial uu. The meter shows the slope you would feel, from steepest down to steepest up.

The viewport now shows the gradient field: a small emerald arrow at every spot of a grid, lying on the landscape, each one the gradient there. On the round bowl they all point straight out from the centre, they grow longer further out where the bowl is steeper, and each one crosses the contour rings at a right angle.

Show the field on the two hills and put the probe in the middle, at x=0x = 0, y=0y = 0.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: gradient and directional derivatives

05Walking downhill: gradient descent

If the gradient points uphill, the opposite arrow, −∇f-\nabla f, points downhill, and it is the steepest way down. That gives a recipe for finding the bottom of a landscape you cannot see all at once: look at the slope under your feet, take a small step the opposite way, look again, repeat. The emerald ball in the viewport is doing exactly this, starting from the probe, and each dot on its trail is one step.

Worked example. At (1,2)(1, 2) the gradient is (2,4)(2, 4). With a step size of η=0.1\eta = 0.1 the step is −0.1×(2,4)=(−0.2, −0.4)-0.1 \times (2, 4) = (-0.2,\ -0.4), which lands at (0.8, 1.6)(0.8,\ 1.6). The height there is 0.64+2.56=3.20.64 + 2.56 = 3.2, down from 55. On this bowl every step multiplies both coordinates by 1−2×0.1=0.81 - 2 \times 0.1 = 0.8, so the ball visits (0.64, 1.28)(0.64,\ 1.28), then (0.512, 1.024)(0.512,\ 1.024), and settles at the bottom after about 29 steps (the HUD's walker readout counts them).

(x,y)new=(x,y)−η ∇f(x,y)(x, y)_{\text{new}} = (x, y) - \tok{step_size}{\eta}\,\tok{gradient}{\nabla f}(x, y)
Gradient descent: one step against the gradient.
Loading figure
Each click takes one step and writes out the arithmetic. The bars underneath show the height after every step.

Walk down the two hills from x=0.4x = 0.4, y=0.3y = 0.3, with step size η=0.2\eta = 0.2.

Same landscape and step size, but start a tiny distance away, at x=0.3x = 0.3, y=0.4y = 0.4.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 7.1

06Step size: too small crawls, too big overshoots

The gradient only describes the ground right under your feet. A tiny step trusts it for a short distance and makes slow progress. A huge step trusts it far beyond where it holds, and can carry you clean over the valley and up the other side.

Worked example. On the round bowl the gradient is (2x,2y)(2x, 2y), so one step turns xx into x−η×2x=(1−2η) xx - \eta \times 2x = (1 - 2\eta)\,x, and the same for yy. Everything depends on that factor 1−2η1 - 2\eta. The viewport is set to η=0.02\eta = 0.02: the factor is 0.960.96, each step only 4% closer, and after 80 steps the ball is still short of the bottom with f=0.007f = 0.007.

xnew=x−η⋅2x=(1−2η) xx_{\text{new}} = x - \tok{step_size}{\eta} \cdot 2x = (1 - 2\tok{step_size}{\eta})\,x
On the round bowl, each step multiplies the position by 1 − 2η.

Set the step size to η=0.5\eta = 0.5.

Now overshoot: set η=0.9\eta = 0.9.

Push it past the limit: set η=1.05\eta = 1.05.

Machine Learning Crash Course, Google, Gradient descent; Learning rate

Compiling scene
Loading editor

Questions learners ask

What is the difference between a derivative and a partial derivative?
An ordinary derivative is the slope of a function with one input. A partial derivative is the slope of a function with several inputs when you change only one of them and hold the others fixed. Geometrically you slice the landscape along one direction and take the slope of the curve you get, so each partial derivative is an ordinary derivative of one slice.
Why does the gradient point in the direction of steepest ascent?
The slope you feel walking in a direction u of length 1 is the dot product of the gradient with u, which equals the length of the gradient times the cosine of the angle between them. The cosine is largest, 1, when the angle is zero, so the slope is largest when you walk along the gradient. It is zero when you walk at a right angle to it, which is the direction of the contour line.
What is the learning rate in gradient descent?
It is the number that multiplies the gradient to set the size of each step, often written as eta. Too small and training crawls; too large and each step overshoots the valley, so the error bounces around or grows. Practitioners usually try a few values spread over several powers of ten and keep the largest one whose error still falls smoothly.

Next in Maths Primer

0.6Probability: chance, averages, spread and BayesRoll dice, draw samples and watch histograms settle: probability, the mean, the variance, the bell curve, and how Bayes' rule updates a belief with evidence.

Sources

  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Machine Learning Crash Course, Google