Skip to content
LearnMLByDoing
Free

0.6 Probability: chance, averages, spread and Bayes

35 minFree lessonbeginnerKhanMML3B1B

Drop thousands of coin flips and dice rolls into a histogram and watch chance turn into dependable numbers: probabilities, the mean, the spread, the bell curve, and what a positive test really tells you.

01Probability is what happens in the long run

Flip a fair coin once and you cannot say what will happen. Flip it a thousand times and you can say a lot: close to half will be heads. The viewport is flipping now. Each cyan cube is one flip falling onto the tails pile (0) or the heads pile (1). Each bar's height is the share of flips so far that landed there, and the thin white ticks mark where the bars should settle: one half each.

Some small numbers. After 10 flips you could easily see 7 heads, a share of 7/10=0.77/10 = 0.7, far from one half. After 1000 flips a share between 0.47 and 0.53 is typical. The share wanders a lot at the start and less and less later on. This settling down is called the law of large numbers, and it is what a probability means: the share of the time something happens if you keep repeating it.

Loading figure
Flip in batches. Early on the emerald line swings far from 0.5; after a few thousand flips it barely moves. Bend the coin with the slider and it settles on the new probability instead.
P(heads)=headsflips  ⟶  12=0.5as the flips pile up\tok{prob}{P}(\text{heads})=\frac{\text{heads}}{\text{flips}}\;\longrightarrow\;\frac12=0.5\quad\text{as the flips pile up}

Slow the flips down to 3 per second and watch the first few land.

Now pour: 300 flips per second.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Statistics and probability: probability basics

02A distribution: the shape of chance

Swap the coin for a die. Now there are six piles, and the white ticks all sit at the same height because a fair die has no favourite face. The full list of outcomes with their probabilities is called a distribution, and a histogram is its picture: one bar per outcome, bar height equal to its probability.

Each face of a fair die has probability 1/6≈0.1671/6 \approx 0.167, and the six bars add up to 6×1/6=16 \times 1/6 = 1, because some face always comes up. To get the chance of a group of outcomes, add their bars. Rolling an even number is P(2)+P(4)+P(6)=3/6=0.5P(2) + P(4) + P(6) = 3/6 = 0.5.

P(X=1)+P(X=2)+⋯+P(X=6)=∑x=16P(X=x)=1\tok{prob}{P}(X=1)+\tok{prob}{P}(X=2)+\dots+\tok{prob}{P}(X=6)=\sum_{x=1}^{6}\tok{prob}{P}(X=x)=1
The bars of any distribution add up to 1.

Load the die so that a six comes up half the time: P(6)=0.5P(6) = 0.5.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.1 and 6.2, probability spaces and discrete distributions

03The mean: where the histogram balances

Picture the bars as blocks stacked on a seesaw. The emerald triangle under the axis is the point where that seesaw balances, and that point is the mean. For the fair die it settles at 3.5, exactly in the middle, even though no roll ever shows 3.5.

A worked example with five numbers, 2, 4, 5, 6 and 8: their total is 25 and there are 5 of them, so the mean is 25/5=525 / 5 = 5. For the die, average the faces using their probabilities as weights: 1⋅16+2⋅16+⋯+6⋅16=216=3.51 \cdot \tfrac16 + 2 \cdot \tfrac16 + \dots + 6 \cdot \tfrac16 = \tfrac{21}{6} = 3.5.

xˉ=1n∑i=1nxi=2+4+5+6+85=5\tok{mean}{\bar x}=\frac{1}{n}\sum_{i=1}^{n}x_i=\frac{2+4+5+6+8}{5}=5
The mean of a list of data.
μ=∑xx P(X=x)=1⋅16+2⋅16+⋯+6⋅16=3.5\mu=\sum_{x}x\,\tok{prob}{P}(X=x)=1\cdot\tfrac16+2\cdot\tfrac16+\dots+6\cdot\tfrac16=3.5
The mean of a distribution: each outcome weighted by its probability.

Load the die again with P(6)=0.5P(6) = 0.5 and watch the balance point.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.4.1, means and expected values

04Spread: variance and standard deviation

Two lists can share a mean and still look very different: 4, 5, 6 hug their mean of 5, while 0, 5, 10 sprawl. Spread measures how far values typically sit from the mean. Drag the points below and watch each one's distance from the balance point.

Loading figure
Each arrow runs from the mean to a point. Drag one point far out and its squared distance takes over the total.

With 2, 4, 5, 6, 8 the distances from the mean 5 are −3,−1,0,+1,+3-3, -1, 0, +1, +3. They add up to 0, because the mean is the balance point and the pulls on each side cancel. So we square them first, which makes every distance count as positive: 9, 1, 0, 1, 9. Their average, 20/5=420 / 5 = 4, is the variance. Its square root, 4=2\sqrt{4} = 2, is the standard deviation: a typical distance from the mean, in the same units as the data.

Variance of 2, 4, 5, 6, 8, one line at a time1/4
  1. 1
    xi−xˉ:−3, −1, 0, 1, 3x_i-\tok{mean}{\bar x}:\quad -3,\ -1,\ 0,\ 1,\ 3

    Distance of each value from the mean 5. They always add up to 0.

In the viewport the amber bracket spans from xˉ−σ\bar x - \sigma to xˉ+σ\bar x + \sigma. For the fair die it settles at σ≈1.71\sigma \approx 1.71 (a variance of 35/12≈2.9235/12 \approx 2.92). For the coin it is exactly 0.5, since every flip, 0 or 1, sits 0.5 away from the mean of 0.5.

Load the die heavily, P(6)=0.9P(6) = 0.9, and watch the bracket.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.4.1 and 6.4.3, variance and empirical statistics

05Add random things and a bell appears

Roll two dice and add them. A total of 7 is now far more common than 2 or 12, because many pairs make 7 and only one pair makes 2. Add more dice and the histogram turns into a smooth, symmetric hill: the bell curve, also called the normal or Gaussian distribution. The amber line is a bell curve with the same mean and spread as the dice.

Two dice have 6×6=366 \times 6 = 36 equally likely pairs. Six of them total 7 (1+6, 2+5, 3+4, 4+3, 5+2, 6+1), so P(7)=6/36≈0.167P(7) = 6/36 \approx 0.167. Only 1+1 totals 2, so P(2)=1/36≈0.028P(2) = 1/36 \approx 0.028. The means add, 3.5+3.5=73.5 + 3.5 = 7, and so do the variances, 2.92+2.92=5.832.92 + 2.92 = 5.83, which gives σ=5.83≈2.42\sigma = \sqrt{5.83} \approx 2.42.

μsum=N×3.5,σsum=N×3512\mu_{\text{sum}}=N\times3.5,\qquad \tok{sigma}{\sigma}_{\text{sum}}=\sqrt{N\times\tfrac{35}{12}}
N dice: the mean grows like N, the spread only like the square root of N.
rolls (one row per roll)[,]4,000 values

In the Python challenge NumPy draws the whole table at once; rolls.sum(axis=1) adds along each row and leaves one total per roll, shape (2000,).

Put the bell curve over a single die: N=1N = 1.

Add five dice together: N=5N = 5.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.5, the Gaussian distribution

06Bayes: what a positive test really tells you

A condition affects 1 person in 100. A test catches 90% of sick people and correctly clears 90% of healthy people. You test positive. How likely is it that you are sick? Many people guess 90%. The viewport answers with 1000 people instead of a formula: the dots first sort by condition (rose for sick), then by test result into a 2 by 2 grid.

Of 1000 people, 10 are sick (1%). The test catches 9 of them. Of the 990 healthy people, 10% test positive anyway: 99 false alarms. So 108 people test positive and only 9 of them are sick, 9/108≈8.3%9 / 108 \approx 8.3\%. The rare condition is swamped by false alarms from the much larger healthy group.

Loading figure
The sliders are linked to the 1000 dots in the viewport. Only the two boxed leaves, everyone who tested positive, matter once you know the result.
P(sick∣+)=sick,+all +\tok{posterior}{P(\text{sick}\mid +)}=\frac{\text{sick},+}{\text{all }+}
Bayes' rule, written as counts over counts.
sick,+:  P(sick) P(+∣sick)=0.009\text{sick},+:\;\tok{prior}{P(\text{sick})}\,\tok{sensitivity}{P(+\mid\text{sick})}=0.009
healthy,+:  P(healthy) P(+∣healthy)=0.099\text{healthy},+:\;P(\text{healthy})\,P(+\mid\text{healthy})=0.099
0.01 × 0.9 = 0.009 and 0.99 × 0.1 = 0.099. P(+ | healthy) is 1 minus the specificity: the false-alarm rate.
P(sick∣+)=0.0090.009+0.099≈0.083\tok{posterior}{P(\text{sick}\mid +)}=\frac{0.009}{0.009+0.099}\approx0.083

Make the condition common: base rate 10%.

Back to a 1% base rate, but with a better test: specificity 99%.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.3, sum rule, product rule and Bayes' theorem

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Statistics and probability: conditional probability and Bayes' theorem

Compiling scene
Loading editor

Questions learners ask

What is the difference between probability and statistics?
Probability starts from a known chance process, like a fair die, and asks what data it will produce. Statistics goes the other way: it starts from data, like a pile of rolls, and asks what process produced it. The law of large numbers links the two, because with enough data the observed shares get close to the true probabilities.
What does standard deviation mean in simple terms?
It is the typical distance of a value from the average. Work it out by finding each value's distance from the mean, squaring those distances, averaging the squares (that average is the variance) and taking the square root. For the numbers 2, 4, 5, 6 and 8 the mean is 5, the variance is 4 and the standard deviation is 2.
Why does a positive result on an accurate test often mean you are probably not sick?
When a condition is rare, the healthy group is so much larger than the sick group that even a small false-positive rate produces more false alarms than true cases. With a 1% base rate and a test that is 90% accurate both ways, 1000 people give 9 true positives and 99 false positives, so only about 8% of positives are sick. Bayes' rule is the arithmetic behind this.

Next in Maths Primer

0.7Exponentials, logarithms, sigmoid and softmaxGrowth that multiplies, logarithms that undo it, and the two squashing functions every classifier ends with: sigmoid for yes or no, softmax for many choices.

Sources

  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)