5.1 Autoencoders, VAEs and latent spaces
An autoencoder squeezes images through a few numbers and rebuilds them; a variational autoencoder also makes those numbers fill a known distribution, so any point you sample decodes to something real.
01Autoencoders: compress, then decompress
An autoencoder is two networks trained as one. The encoder squeezes an input into a short code ; the decoder rebuilds from that code alone. Training asks for . Because the code is much smaller than the input, the network cannot copy pixels through: it has to find the few numbers that describe what varies across the data.
Each 8×8 glyph is 64 numbers. The encoder maps x of shape (B, 64) to z of shape (B, 2), and the decoder maps z back to x̂ of shape (B, 64): the code is 32 times smaller than the image.
The data in this lab are 48 tiny images of one pen stroke: 8 short, 16 medium and 24 long strokes, each set at evenly spaced rotations. The figure lets you run the autoencoder on any of them. Pick a glyph, read its two code numbers, and check that the decoder brings it back.
The viewport shows the same latent plane as the floor. Each cyan dot is one glyph's code, and the screen floating above the emerald probe ring shows the decoder's output for the code under it. While the scene plays, the probe tours the plane: on the rings the screen shows a clean stroke that turns as the probe moves round, and between the rings it shows noise.
The link to PCA. Lesson 2.7 projected data onto its top principal directions and reconstructed it from those numbers. That is an autoencoder with a linear encoder and a linear decoder: trained with squared error, a linear autoencoder with code units learns the same subspace as PCA. Nonlinear encoders and decoders can bend the code space around curved data, like the circle of rotations here, which is what makes them worth training.
Hold the probe on the code of the long stroke at 45°, , with a plain autoencoder ().
Deep Learning, Goodfellow, Bengio & Courville, Ch. 14.1 (undercomplete autoencoders)
Probabilistic Machine Learning: An Introduction, Kevin P. Murphy, Ch. 20.3 (autoencoders)
02Why a plain autoencoder does not generate
To generate a new image you need a way to pick a code without an input, then decode it. The natural plan is to sample from a simple distribution such as . The viewport draws 256 such samples on the plane, emerald where the decoder was trained and rose where it was not.
Only 86 of the 256 samples (34%) land close enough to one of the 48 codes. The rest fall into holes: the gaps between the rings, the middle, and the space between neighbouring codes on a ring. The decoder was only ever asked to reconstruct glyphs from the 48 exact codes, so the loss says nothing about any other . The rose tint on the floor marks where the prior puts samples but the decoder was never trained.
Hold the probe at , between the short-stroke ring and the medium-stroke ring.
Two things are missing. First, the decoder never sees neighbourhoods of codes, only points, so there is no pressure for nearby codes to decode to similar images. Second, nothing ties the codes to the distribution you want to sample from: the encoder could put its rings at radius 100 or squeeze them into a corner, and the reconstruction loss would not change. A variational autoencoder fixes both: it makes the encoder output a small cloud instead of a point, and it charges a price for clouds that stray from .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 20.10.3
03The VAE: a probabilistic encoder, a prior and a decoder
A variational autoencoder (VAE) starts from a generative story. First draw a code from the prior, then draw an image from the decoder's distribution:
Training needs the reverse direction: which codes could have produced this ? The exact posterior is intractable, so an encoder network approximates it with a diagonal Gaussian. It outputs two vectors per input, a mean and a log-variance:
Two heads of the same shape as z. The network predicts log σ² rather than σ because it can take any real value, while σ must stay positive.
The viewport is at , the standard VAE. Every code has become a cloud: the ellipses have and the means have moved in, to 93.5% of their old radius along and 86.6% along . The decoder is trained on samples from these clouds, so it learns what to draw everywhere they overlap, and the clouds overlap enough to tile the prior. On the floor the rose holes are gone.
Hold the probe at again, the hole from the previous step, with .
Auto-Encoding Variational Bayes, Kingma & Welling, 2014, Sec. 2 and 3
Probabilistic Machine Learning: An Introduction, Kevin P. Murphy, Ch. 20.3.5
The rest of this lesson is in Pro
- 04The ELBO: reconstruction minus KL
- 05The reparameterisation trick
- 06β-VAE, posterior collapse and interpolation
- 07VAEs among the generative model families
- Python challenge: The VAE loss: Gaussian KL, reparameterisation and the ELBO
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between an autoencoder and a variational autoencoder?
- An autoencoder maps each input to a single code and is trained only to reconstruct, so the space between codes is undefined and you cannot sample new data from it reliably. A variational autoencoder maps each input to a distribution over codes, trains the decoder on samples from those distributions, and adds a KL penalty that pulls them toward a fixed prior such as a standard normal. The result is a latent space you can sample from and interpolate in.
- Why are VAE samples blurry?
- When the posteriors of different images overlap, the decoder that maximises the likelihood outputs the average of all images that could have produced that code, and averages of slightly different images look soft. A Gaussian or Bernoulli pixel likelihood also rewards being close on average rather than sharp. Lower KL weights, more expressive decoders, perceptual or adversarial losses, and using the VAE only as a compressor for a diffusion model are the usual fixes.
- What is the ELBO in simple terms?
- The evidence lower bound is a quantity you can compute that is never larger than the log-likelihood of the data, which you cannot compute. It has two parts: how well the decoder reconstructs the input from codes drawn from the encoder, minus how far the encoder's distribution is from the prior. Maximising it trains the encoder and decoder together and pushes up the true likelihood.
Next in Generative AI and Diffusion
5.2Score matching and Langevin dynamicsThe gradient of log-density as a vector field, and sampling by following it with injected noise.Sources
- Deep Learning, Goodfellow, Bengio & Courville
- Auto-Encoding Variational Bayes, Kingma & Welling, 2014
- Probabilistic Machine Learning: An Introduction, Kevin P. Murphy
- Dive into Deep Learning, Zhang, Lipton, Li & Smola
- Maths, CS & AI Compendium, Henry Ndubuaku