0.7 Exponentials, logarithms, sigmoid and softmax
Four small functions do most of the work at the end of every classifier: the exponential grows by multiplying, the logarithm counts the multiplications, and sigmoid and softmax turn raw scores into probabilities you can trust to add up to 1.
01Growing by multiplying
Fold a sheet of paper and it is 2 layers thick. Fold again: 4. Again: 8. Each fold multiplies the thickness by 2 instead of adding a fixed amount, and that is what exponential growth means. The emerald curve in the viewport is this rule drawn for every input, and the emerald point slides along it.
Read the grey dots at the whole numbers : their heights are . Each step right multiplies the height by 2. Going left divides by 2 instead, so sits at and at . The curve never touches zero: halving a positive number always leaves something positive.
Pause on with base .
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 1: Exponential growth and decay
02e: the growth whose slope equals its height
An exponential curve gets steeper exactly as fast as it gets taller: twice as high means twice as steep. The amber line touching the curve at the emerald point is the tangent, the straight line with the curve's slope at that spot. The amber triangle under it goes 1 step left and measures the slope as its rise. The cyan stick next to the rise is the height.
For base 2 the rise is always of the height, wherever the point is. For base 3 it is of the height. Somewhere between 2 and 3 there is a base where the rise and the height are equal everywhere. That base is the number , and the viewport is set to it now. Wherever the point slides, the telemetry reads slope ÷ height and the triangle's bottom corner rests exactly on the horizontal axis. At , for example, the height and the slope are both , and the tangent meets the axis one step to the left, at .
Compare base at the same spot, , paused.
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Differential calculus: derivatives of e^x and ln(x)
03Logarithms: how many times do I multiply?
Now ask the question backwards: how many times must I multiply 2 by itself to reach 8? Watch the emerald point hop along the amber curve. Each hop multiplies its position on the number line (the cyan dot) by 2, , while the amber dot on the vertical axis counts the hops: 3. So the logarithm of 8 in base 2 is 3.
Some worked answers with base 2: , (zero hops, you start at 1), (one hop backwards, which divides). Answers need not be whole numbers: , four full hops reach 16 and a part-hop covers the rest.
Exponential and logarithm undo each other. Hop times from 1 and you land on ; the telemetry row always reads back the input. This is also where step 2's ratio came from: the slope ÷ height of is , and .
Reach in base 2.
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 2: Logarithms
04Logs turn multiplying into adding
Hops add up. Reaching 16 in base 2 takes 4 hops, and : 1 hop to reach 2, then 3 more hops multiply by 8. Multiplying the numbers added the hop counts. The ruler below shows the same thing for any two numbers: it is spaced by logarithms but labelled with the numbers, so laying two lengths end to end multiplies.
- 1
Write each number as e to some power; the powers are their logs.
This matters because models multiply probabilities. The chance of a 1000-word sentence is a product of 1000 word probabilities. Even if each is , the product is , and slightly smaller numbers fall below what a computer can store and become exactly 0. The sum of the logs is an ordinary number: .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 8.3 (log-likelihood)
05Sigmoid: any number becomes a probability
A spam filter adds up evidence into a single score : large and positive means probably spam, large and negative means probably fine. A score can be any number, but a probability must lie between 0 and 1. In the viewport, cyan dots start all along the number line, rise to the cyan S-curve and slide right onto the emerald rail: whatever goes in, it lands between 0 and 1.
Three worked values. At : , so , a coin flip. At : , so . At : , so . Notice : flipping the sign of the score swaps "yes" and "no".
Pause the probe at , then push it to the far end, .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.10 (logistic sigmoid)
06Softmax: many scores become probabilities
A photo classifier gives five scores, one per class: cat , dog , fox , owl , bee . We want five probabilities that are positive and add up to 1, with the biggest score getting the biggest share. Watch the viewport run the recipe stage by stage, named along the top: the cyan scores become amber exponentials (all positive now, the owl bar flips up), the exponentials stack into one sum column, and dividing everything by that sum leaves emerald probabilities with a total of exactly 1.
Worked numbers, as the bar labels show them: , , , , . They add up to . Dividing gives cat , dog , fox , owl , bee .
One score in, one probability out, per class. A real image model does the same over 1000 classes.
- 1
e^(a − m) = e^a · e^(−m): the same factor appears on top and in every term of the sum.
Raise the dog's score from 1 to 2, level with the cat, and look at the finished probabilities.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.1 (overflow and underflow) and 6.2.2.3 (softmax units)
07Temperature and surprise
Temperature divides every score before the exponential. A small stretches the gaps between scores, so the winner takes almost everything. A large squeezes the gaps, so the probabilities even out. With our five scores: gives the cat , gives , gives .
Training needs one number that says how wrong a prediction was. Take the probability the model gave the correct answer, , and measure its surprise . Correct class cat with : surprise . If the photo were really an owl, and the surprise is . The rose column in the viewport is this number, the cross-entropy loss.
Sharpen the softmax with .
Keep but make the owl the correct class.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.2.1.1 (cross-entropy as negative log-likelihood)
Questions learners ask
- What is the difference between sigmoid and softmax?
- Sigmoid takes one score and returns one probability, the chance of yes, with no as one minus it. Softmax takes a list of scores, one per option, and returns a list of probabilities that add up to 1. With exactly two options, softmax of the scores z and 0 gives the same answer as the sigmoid of z, so sigmoid is the two-class special case.
- Why does machine learning use e and the natural log instead of base 2 or base 10?
- Any base would work, because changing the base only multiplies every log by a constant. Base e is chosen because the slope of e to the x is e to the x itself, so derivatives of exponentials and logs come out without extra constants. That keeps the formulas used in training, such as the gradient of the cross-entropy loss, as short as possible.
- What does temperature do in a language model?
- Before choosing the next word, a language model turns its scores for every possible word into probabilities with a softmax, after dividing the scores by the temperature. A temperature below 1 sharpens the probabilities so the most likely words dominate and the output becomes more predictable. A temperature above 1 flattens them so less likely words get chosen more often and the output becomes more varied.
Sources
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
- Deep Learning, Goodfellow, Bengio & Courville