Skip to content
LearnMLByDoing
Free

0.7 Exponentials, logarithms, sigmoid and softmax

30 minFree lessonbeginnerKhanMML3B1BDL

Four small functions do most of the work at the end of every classifier: the exponential grows by multiplying, the logarithm counts the multiplications, and sigmoid and softmax turn raw scores into probabilities you can trust to add up to 1.

01Growing by multiplying

Fold a sheet of paper and it is 2 layers thick. Fold again: 4. Again: 8. Each fold multiplies the thickness by 2 instead of adding a fixed amount, and that is what exponential growth means. The emerald curve in the viewport is this rule drawn for every input, and the emerald point slides along it.

Read the grey dots at the whole numbers x=0,1,2,3x = 0, 1, 2, 3: their heights are 1,2,4,81, 2, 4, 8. Each step right multiplies the height by 2. Going left divides by 2 instead, so x=−1x = -1 sits at 0.50.5 and x=−2x = -2 at 0.250.25. The curve never touches zero: halving a positive number always leaves something positive.

bx=b⋅b⋯b⏟x times,23=2⋅2⋅2=8\tok{base}{b}^{x} = \underbrace{\tok{base}{b}\cdot \tok{base}{b}\cdots \tok{base}{b}}_{x\text{ times}},\qquad 2^3 = 2\cdot2\cdot2 = 8
The base b is the number being multiplied; the exponent x counts the multiplications.

Pause on x=3x = 3 with base b=2b = 2.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 1: Exponential growth and decay

02e: the growth whose slope equals its height

An exponential curve gets steeper exactly as fast as it gets taller: twice as high means twice as steep. The amber line touching the curve at the emerald point is the tangent, the straight line with the curve's slope at that spot. The amber triangle under it goes 1 step left and measures the slope as its rise. The cyan stick next to the rise is the height.

For base 2 the rise is always 0.6930.693 of the height, wherever the point is. For base 3 it is 1.0991.099 of the height. Somewhere between 2 and 3 there is a base where the rise and the height are equal everywhere. That base is the number e≈2.718e \approx 2.718, and the viewport is set to it now. Wherever the point slides, the telemetry reads slope ÷ height =1.000= 1.000 and the triangle's bottom corner rests exactly on the horizontal axis. At x=1x = 1, for example, the height and the slope are both e1=2.718e^1 = 2.718, and the tangent meets the axis one step to the left, at x=0x = 0.

slope of ex at x  =  exe.g. at x=1: slope=height=2.718\text{slope of } \tok{exp}{e^{x}} \text{ at } x \;=\; \tok{exp}{e^{x}} \qquad\text{e.g. at } x=1:\ \text{slope} = \text{height} = 2.718
The telemetry row "slope ÷ height" reads 1.000 for base e.

Compare base b=2b = 2 at the same spot, x=1x = 1, paused.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Differential calculus: derivatives of e^x and ln(x)

03Logarithms: how many times do I multiply?

Now ask the question backwards: how many times must I multiply 2 by itself to reach 8? Watch the emerald point hop along the amber curve. Each hop multiplies its position on the number line (the cyan dot) by 2, 1→2→4→81 \to 2 \to 4 \to 8, while the amber dot on the vertical axis counts the hops: 3. So the logarithm of 8 in base 2 is 3.

log⁡ba=number of times you multiply b to reach a,log⁡28=3 because 23=8\log_{\tok{base}{b}} a = \text{number of times you multiply } \tok{base}{b} \text{ to reach } a,\qquad \log_2 8 = 3 \ \text{because}\ 2^3 = 8

Some worked answers with base 2: log⁡216=4\log_2 16 = 4, log⁡21=0\log_2 1 = 0 (zero hops, you start at 1), log⁡20.5=−1\log_2 0.5 = -1 (one hop backwards, which divides). Answers need not be whole numbers: log⁡220=4.32\log_2 20 = 4.32, four full hops reach 16 and a part-hop covers the rest.

Exponential and logarithm undo each other. Hop log⁡ba\log_b a times from 1 and you land on aa; the telemetry row b(log⁡ba)b^{(\log_b a)} always reads back the input. This is also where step 2's ratio came from: the slope ÷ height of bxb^x is ln⁡ b\tok{log}{\ln}\,\tok{base}{b}, and ln⁡2=0.693\ln 2 = 0.693.

Reach a=20a = 20 in base 2.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 2: Logarithms

04Logs turn multiplying into adding

Hops add up. Reaching 16 in base 2 takes 4 hops, and 16=2×816 = 2 \times 8: 1 hop to reach 2, then 3 more hops multiply by 8. Multiplying the numbers added the hop counts. The ruler below shows the same thing for any two numbers: it is spaced by logarithms but labelled with the numbers, so laying two lengths end to end multiplies.

Loading figure
With a=2a = 2 and b=4b = 4: ln⁡2+ln⁡4=0.693+1.386=2.079=ln⁡8\ln 2 + \ln 4 = 0.693 + 1.386 = 2.079 = \ln 8. Try the 0.5 × 0.2 example: both logs are negative and the product, 0.1, lands further left.
Why the rule always holds1/3
  1. 1
    a=eu,b=ev(u=ln⁡a, v=ln⁡b)a = e^{u},\quad b = e^{v}\qquad (u = \ln a,\ v = \ln b)

    Write each number as e to some power; the powers are their logs.

This matters because models multiply probabilities. The chance of a 1000-word sentence is a product of 1000 word probabilities. Even if each is 0.90.9, the product is 0.91000≈1.7×10−460.9^{1000} \approx 1.7\times10^{-46}, and slightly smaller numbers fall below what a computer can store and become exactly 0. The sum of the logs is an ordinary number: 1000×ln⁡0.9=−105.41000 \times \ln 0.9 = -105.4.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 8.3 (log-likelihood)

05Sigmoid: any number becomes a probability

A spam filter adds up evidence into a single score zz: large and positive means probably spam, large and negative means probably fine. A score can be any number, but a probability must lie between 0 and 1. In the viewport, cyan dots start all along the number line, rise to the cyan S-curve and slide right onto the emerald rail: whatever goes in, it lands between 0 and 1.

σ(z)=11+e−z\tok{sigmoid}{\sigma}(z) = \frac{1}{1+\tok{exp}{e^{-z}}}
Big positive z: e^(−z) is tiny and σ is close to 1. Big negative z: e^(−z) is huge and σ is close to 0.

Three worked values. At z=0z = 0: e0=1e^{0} = 1, so σ=1/(1+1)=0.5\sigma = 1/(1+1) = 0.5, a coin flip. At z=2z = 2: e−2=0.135e^{-2} = 0.135, so σ=1/1.135=0.881\sigma = 1/1.135 = 0.881. At z=−2z = -2: e2=7.389e^{2} = 7.389, so σ=1/8.389=0.119\sigma = 1/8.389 = 0.119. Notice 0.881+0.119=10.881 + 0.119 = 1: flipping the sign of the score swaps "yes" and "no".

Pause the probe at z=2z = 2, then push it to the far end, z=6z = 6.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.10 (logistic sigmoid)

06Softmax: many scores become probabilities

A photo classifier gives five scores, one per class: cat 22, dog 11, fox 0.50.5, owl −1-1, bee 00. We want five probabilities that are positive and add up to 1, with the biggest score getting the biggest share. Watch the viewport run the recipe stage by stage, named along the top: the cyan scores become amber exponentials (all positive now, the owl bar flips up), the exponentials stack into one sum column, and dividing everything by that sum leaves emerald probabilities with a total of exactly 1.

pi=ezi∑jezj\tok{prob}{p_i} = \frac{\tok{exp}{e^{z_i}}}{\tok{sum}{\sum_j} e^{z_j}}
Exponentiate each score, then divide by the total of all of them.

Worked numbers, as the bar labels show them: e2=7.389e^{2} = 7.389, e1=2.718e^{1} = 2.718, e0.5=1.649e^{0.5} = 1.649, e−1=0.368e^{-1} = 0.368, e0=1e^{0} = 1. They add up to 13.12413.124. Dividing gives cat 7.389/13.124=0.5637.389/13.124 = 0.563, dog 0.2070.207, fox 0.1260.126, owl 0.0280.028, bee 0.0760.076.

z (scores) → p (probabilities)[]5 values

One score in, one probability out, per class. A real image model does the same over 1000 classes.

Loading figure
Three of the classes, every arithmetic step visible. Add 1000 to every score with the max trick off and the exponentials overflow; switch the trick on and the answer comes back unchanged.
Subtracting the largest score changes nothing1/2
  1. 1
    ezi−m∑jezj−m=ezi e−me−m∑jezj\frac{e^{z_i - m}}{\sum_j e^{z_j - m}} = \frac{e^{z_i}\, e^{-m}}{e^{-m}\sum_j e^{z_j}}

    e^(a − m) = e^a · e^(−m): the same factor appears on top and in every term of the sum.

Raise the dog's score from 1 to 2, level with the cat, and look at the finished probabilities.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.1 (overflow and underflow) and 6.2.2.3 (softmax units)

07Temperature and surprise

Temperature TT divides every score before the exponential. A small TT stretches the gaps between scores, so the winner takes almost everything. A large TT squeezes the gaps, so the probabilities even out. With our five scores: T=0.5T = 0.5 gives the cat 0.8290.829, T=1T = 1 gives 0.5630.563, T=2T = 2 gives 0.3750.375.

pi=ezi/T∑jezj/T\tok{prob}{p_i} = \frac{e^{z_i/\tok{temp}{T}}}{\sum_j e^{z_j/\tok{temp}{T}}}
Loading figure
This slider is linked to the viewport's TT. Entropy measures how spread out the probabilities are: it falls towards 0 as TT shrinks and climbs towards the uniform value, 2.322.32 bits for 5 classes, as TT grows.

Training needs one number that says how wrong a prediction was. Take the probability the model gave the correct answer, pyp_y, and measure its surprise −ln⁡py-\ln p_y. Correct class cat with p=0.563p = 0.563: surprise −ln⁡0.563=0.574-\ln 0.563 = 0.574. If the photo were really an owl, p=0.028p = 0.028 and the surprise is 3.5743.574. The rose column in the viewport is this number, the cross-entropy loss.

−ln⁡pypy=1⇒0,py=0.5⇒0.693,py=0.01⇒4.605\tok{ce}{-\ln p_y}\qquad p_y = 1 \Rightarrow 0,\quad p_y = 0.5 \Rightarrow 0.693,\quad p_y = 0.01 \Rightarrow 4.605
Loading figure
Confident and right costs almost nothing; confident and wrong costs a lot. Halving pp always adds the same 0.6930.693.

Sharpen the softmax with T=0.5T = 0.5.

Keep T=1T = 1 but make the owl the correct class.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.2.1.1 (cross-entropy as negative log-likelihood)

Compiling scene
Loading editor

Questions learners ask

What is the difference between sigmoid and softmax?
Sigmoid takes one score and returns one probability, the chance of yes, with no as one minus it. Softmax takes a list of scores, one per option, and returns a list of probabilities that add up to 1. With exactly two options, softmax of the scores z and 0 gives the same answer as the sigmoid of z, so sigmoid is the two-class special case.
Why does machine learning use e and the natural log instead of base 2 or base 10?
Any base would work, because changing the base only multiplies every log by a constant. Base e is chosen because the slope of e to the x is e to the x itself, so derivatives of exponentials and logs come out without extra constants. That keeps the formulas used in training, such as the gradient of the cross-entropy loss, as short as possible.
What does temperature do in a language model?
Before choosing the next word, a language model turns its scores for every possible word into probabilities with a softmax, after dividing the scores by the temperature. A temperature below 1 sharpens the probabilities so the most likely words dominate and the output becomes more predictable. A temperature above 1 flattens them so less likely words get chosen more often and the output becomes more varied.

Sources

  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Deep Learning, Goodfellow, Bengio & Courville