2.1 Gradient descent, saddle points and optimizer dynamics
An optimizer turns the local slope of the loss into a step, and SGD, momentum and Adam turn the same slope into very different paths across a bowl, a saddle and a curved valley.
01The loss is a landscape
Training a model means choosing parameters that make a scalar loss small. With two parameters the loss is a surface: and span the floor and is the height. Every optimizer in this lesson sees only one thing about that surface at a time: its slope at the current point.
Two parameters so the loss can be drawn. A real network has millions, and every update below is applied elementwise to that whole vector.
The slope is the gradient , the direction of steepest ascent. Near the surface is well approximated by its tangent plane, the grey patch under the emerald ball:
The surface here is the convex bowl . Its gradient is , so at the start it is , and the steepest way down, , points mostly along rather than at the minimum at the origin. The bowl is four times steeper along than along , and that mismatch is what the rest of the lesson is about.
Three optimizers start from : SGD in emerald, momentum in amber and Adam in cyan. Each dot is one step, so dot spacing is step size. The emerald ball is labelled SGD, as the plain update is in deep-learning libraries, but with the cockpit's gradient noise at 0 it uses the exact gradient: it is full-batch gradient descent, with no minibatch noise, so every run is deterministic. Step 3 turns the noise on.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.1 and 5.8
Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3
02Gradient descent and the learning rate
Gradient descent moves a small distance against the gradient. The learning rate sets how far:
The viewport is at , so the first step from is , landing at : moved a little toward 0, but jumped right past 0 to the other side. To see when this converges, the derivation below needs one more object. The Hessian is the table of second derivatives, how fast the slope itself changes. For this bowl it is diagonal: and , the curvature along each axis. Its eigenvalues are those curvatures, 1 and 4, and its eigen-directions are the two axes.
- 1
Put δ = −ηg into the Taylor model. To first order the loss can only fall, but the model is only trusted for small η.
Because is diagonal, the bowl is two separate one-dimensional problems, one per axis, each a parabola . The figure shows one of them. Every step multiplies by the same factor , so the iterates hop across the parabola: on the same side while the factor is positive, from side to side when it is negative, and further out each time once . Its learning-rate slider is the cockpit's , so the 3D race follows it.
Set , the learning rate that makes the factor exactly .
Set , exactly the stability limit.
Cross the stability limit: set , just above , on the same bowl and start.
The slow direction sets the pace. Along the factor is , so a learning rate small enough for the steep axis makes the shallow axis crawl. The ratio , the condition number, measures how bad this trade-off is; it is 4 here and can be in the thousands for real networks.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3.1
Machine Learning Crash Course, Google, Gradient descent; Learning rate
03Saddle points: zero gradient, not a minimum
A critical point has . The Hessian decides what it is: all eigenvalues positive is a minimum, mixed signs is a saddle. The surface now is , with Hessian at the origin and two minima at where . All three optimizers start at , almost exactly on the ridge.
With SGD needs about steps to grow from to . Meanwhile the tangent plane is nearly flat, the HUD shows close to zero, and the loss sits at , well above the minimum of . A small gradient at a loss that is not the best you have seen is the signature of a saddle, not of convergence.
In high-dimensional networks, critical points with high loss are overwhelmingly saddles rather than local minima, so long plateaus are a more common failure than being truly trapped. Minibatch noise and the adaptive methods below are the practical remedies. (On this surface and on the Rosenbrock valley the drawn height is log-compressed so the low region stays visible; the HUD values are the true .)
Start at exactly instead of .
Real training does not use exact gradients. A minibatch of examples gives an estimate whose noise shrinks roughly like , so small batches mean noisy steps. The cockpit's gradient noise slider adds seeded Gaussian noise of that kind to every optimizer's gradient (the same seed on every restart, so a run repeats exactly). Above 0 the emerald ball really is stochastic gradient descent.
Same ridge start, now with gradient noise .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2.3
04Momentum: velocity across a ravine
The surface is now a scaled Rosenbrock function, , a curved valley with its minimum at . At the minimum the Hessian eigenvalues are about across the valley and along it, a condition number of about 133. SGD drops into the valley and then crawls along its floor.
- 1
Unrolled, the velocity is an exponentially weighted sum of past gradients.
Both effects help on the valley: the cross-valley gradient alternates and is averaged away, while the small, consistent along-valley gradient is amplified. Hover the velocity symbol to see the amber velocity arrow, drawn three times longer than the actual step. (The valley's drawn height is log-compressed so its floor stays visible; the HUD shows the true .)
The sum is an exponential moving average in disguise. Write , starting from ; unrolling gives , so the velocity is . The figure feeds a noisy gradient with mean 1 into that average. Its slider is the cockpit's momentum.
Move to the bowl with a gentler momentum, .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.3.2
05Adam: per-coordinate step sizes
Adam keeps two exponential moving averages per parameter: one of the gradient, which estimates the direction, and one of the squared gradient, which estimates the scale. It then takes a step of roughly in every coordinate, shrinking it where the gradient is noisy or changes sign.
- 1
m starts at 0, so it is a weighted sum whose weights add up to less than 1.
Three steps with numbers make this concrete. Take a constant gradient with the default , :
The figure runs the same averages on a noisy gradient. Its is the cockpit's, so the cyan ball in the viewport uses the value you set here.
One first and one second moment per parameter, the same shape as θ. For a large model, Adam's state is two extra copies of the weights.
Adam is two older ideas combined: the first moment is momentum, and dividing by is RMSProp. Setting removes the momentum part, so Adam becomes RMSProp with bias correction on .
Set Adam's on the bowl.
Adam: A Method for Stochastic Optimization, Kingma & Ba, 2015, Sec. 2 and 3
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.5.3
06Reading a trajectory
Loss curves tell you that something is wrong; trajectories tell you what. Three signatures cover most problems:
Divergence. Steps alternate sides of the valley and get longer, the dots spread apart, and rises. The cause is . Lower below .
Saddle stall. Dots bunch up, is close to zero, yet is above the best value, and then the path suddenly peels away. Use momentum, an adaptive method, or minibatch noise.
Ill-conditioned crawl. Steady, tiny steps along a curved valley floor; the loss falls slowly but never rises. Use momentum or per-coordinate scaling, or reparameterize so the curvature is more even.
Schedules, clipping and AdamW. Three standard tools sit next to "lower ". A learning-rate schedule changes during training: a short warm-up from a small value lets Adam's estimates settle before full-size steps, and a step or cosine decay at the end shrinks the steps so the noisy jitter around the minimum dies down (turn on gradient noise in the cockpit to see that jitter). Gradient clipping rescales the gradient to norm whenever : the direction is kept, but one exploding gradient can no longer throw the parameters off the map. AdamW applies weight decay directly to the weights, , instead of adding to the loss, where Adam's division by would weaken the penalty on weights with large gradients. Lesson 2.3 covers regularisation itself.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2
Questions learners ask
- What is the difference between gradient descent and stochastic gradient descent?
- Gradient descent uses the exact gradient of the full training loss. Stochastic gradient descent estimates it from a random minibatch, which is much cheaper per step and adds noise. The update rule is the same. In this lesson the runs are exact gradient descent until you raise the gradient-noise slider, which adds minibatch-like noise; that noise is also what helps SGD leave saddle points.
- How do I choose a learning rate?
- On a quadratic, gradient descent is stable only when the learning rate is below 2 divided by the largest Hessian eigenvalue, and the smallest eigenvalue then sets how slowly it converges. In practice, sweep the learning rate on a log scale for a short run, pick the largest value whose loss falls smoothly, and add warm-up and decay. For Adam on neural networks, 0.001 is a common starting point.
- Why does Adam need bias correction?
- Adam's moment estimates start at zero, so during the first steps they are shrunk by a factor of 1 minus beta to the power t. Dividing by that factor makes the early steps the intended size. Without it, the first step with the default betas is about 3.16 times the learning rate.
Next in Classical Machine Learning
2.2Linear and logistic regression from first principlesClosed-form least squares as a projection, then the sigmoid, cross-entropy and the convex surface it creates.Sources
- Machine Learning Crash Course, Google
- Deep Learning, Goodfellow, Bengio & Courville
- Adam: A Method for Stochastic Optimization, Kingma & Ba, 2015
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong