0.6 Probability: chance, averages, spread and Bayes
Drop thousands of coin flips and dice rolls into a histogram and watch chance turn into dependable numbers: probabilities, the mean, the spread, the bell curve, and what a positive test really tells you.
01Probability is what happens in the long run
Flip a fair coin once and you cannot say what will happen. Flip it a thousand times and you can say a lot: close to half will be heads. The viewport is flipping now. Each cyan cube is one flip falling onto the tails pile (0) or the heads pile (1). Each bar's height is the share of flips so far that landed there, and the thin white ticks mark where the bars should settle: one half each.
Some small numbers. After 10 flips you could easily see 7 heads, a share of , far from one half. After 1000 flips a share between 0.47 and 0.53 is typical. The share wanders a lot at the start and less and less later on. This settling down is called the law of large numbers, and it is what a probability means: the share of the time something happens if you keep repeating it.
Slow the flips down to 3 per second and watch the first few land.
Now pour: 300 flips per second.
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Statistics and probability: probability basics
02A distribution: the shape of chance
Swap the coin for a die. Now there are six piles, and the white ticks all sit at the same height because a fair die has no favourite face. The full list of outcomes with their probabilities is called a distribution, and a histogram is its picture: one bar per outcome, bar height equal to its probability.
Each face of a fair die has probability , and the six bars add up to , because some face always comes up. To get the chance of a group of outcomes, add their bars. Rolling an even number is .
Load the die so that a six comes up half the time: .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.1 and 6.2, probability spaces and discrete distributions
03The mean: where the histogram balances
Picture the bars as blocks stacked on a seesaw. The emerald triangle under the axis is the point where that seesaw balances, and that point is the mean. For the fair die it settles at 3.5, exactly in the middle, even though no roll ever shows 3.5.
A worked example with five numbers, 2, 4, 5, 6 and 8: their total is 25 and there are 5 of them, so the mean is . For the die, average the faces using their probabilities as weights: .
Load the die again with and watch the balance point.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.4.1, means and expected values
04Spread: variance and standard deviation
Two lists can share a mean and still look very different: 4, 5, 6 hug their mean of 5, while 0, 5, 10 sprawl. Spread measures how far values typically sit from the mean. Drag the points below and watch each one's distance from the balance point.
With 2, 4, 5, 6, 8 the distances from the mean 5 are . They add up to 0, because the mean is the balance point and the pulls on each side cancel. So we square them first, which makes every distance count as positive: 9, 1, 0, 1, 9. Their average, , is the variance. Its square root, , is the standard deviation: a typical distance from the mean, in the same units as the data.
- 1
Distance of each value from the mean 5. They always add up to 0.
In the viewport the amber bracket spans from to . For the fair die it settles at (a variance of ). For the coin it is exactly 0.5, since every flip, 0 or 1, sits 0.5 away from the mean of 0.5.
Load the die heavily, , and watch the bracket.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.4.1 and 6.4.3, variance and empirical statistics
05Add random things and a bell appears
Roll two dice and add them. A total of 7 is now far more common than 2 or 12, because many pairs make 7 and only one pair makes 2. Add more dice and the histogram turns into a smooth, symmetric hill: the bell curve, also called the normal or Gaussian distribution. The amber line is a bell curve with the same mean and spread as the dice.
Two dice have equally likely pairs. Six of them total 7 (1+6, 2+5, 3+4, 4+3, 5+2, 6+1), so . Only 1+1 totals 2, so . The means add, , and so do the variances, , which gives .
In the Python challenge NumPy draws the whole table at once; rolls.sum(axis=1) adds along each row and leaves one total per roll, shape (2000,).
Put the bell curve over a single die: .
Add five dice together: .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.5, the Gaussian distribution
06Bayes: what a positive test really tells you
A condition affects 1 person in 100. A test catches 90% of sick people and correctly clears 90% of healthy people. You test positive. How likely is it that you are sick? Many people guess 90%. The viewport answers with 1000 people instead of a formula: the dots first sort by condition (rose for sick), then by test result into a 2 by 2 grid.
Of 1000 people, 10 are sick (1%). The test catches 9 of them. Of the 990 healthy people, 10% test positive anyway: 99 false alarms. So 108 people test positive and only 9 of them are sick, . The rare condition is swamped by false alarms from the much larger healthy group.
Make the condition common: base rate 10%.
Back to a 1% base rate, but with a better test: specificity 99%.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 6.3, sum rule, product rule and Bayes' theorem
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Statistics and probability: conditional probability and Bayes' theorem
Questions learners ask
- What is the difference between probability and statistics?
- Probability starts from a known chance process, like a fair die, and asks what data it will produce. Statistics goes the other way: it starts from data, like a pile of rolls, and asks what process produced it. The law of large numbers links the two, because with enough data the observed shares get close to the true probabilities.
- What does standard deviation mean in simple terms?
- It is the typical distance of a value from the average. Work it out by finding each value's distance from the mean, squaring those distances, averaging the squares (that average is the variance) and taking the square root. For the numbers 2, 4, 5, 6 and 8 the mean is 5, the variance is 4 and the standard deviation is 2.
- Why does a positive result on an accurate test often mean you are probably not sick?
- When a condition is rare, the healthy group is so much larger than the sick group that even a small false-positive rate produces more false alarms than true cases. With a 1% base rate and a test that is 90% accurate both ways, 1000 people give 9 true positives and 99 false positives, so only about 8% of positives are sick. Bayes' rule is the arithmetic behind this.
Sources
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)