Textbook of AI

Also known as: VAE

A Variational Autoencoder (VAE), introduced by Kingma and Welling in 2013, is a probabilistic generative model that imposes a structured latent space on the autoencoder framework. Rather than encoding an input to a single latent point, the encoder outputs the parameters of a distribution (typically Gaussian with mean $\mu$ and diagonal covariance $\sigma^2 I$), and the latent code is sampled from this distribution. The loss is the Evidence Lower Bound (ELBO):

$$\mathcal{L} = \mathbb{E}{q(z|x)}[\log p(x|z)] - D{KL}(q(z|x) \parallel p(z))$$

The first term is a reconstruction loss; the second regularises the encoder's distribution toward a prior (typically a standard Gaussian).

The reparameterisation trick is the key technical innovation. Sampling $z \sim \mathcal{N}(\mu, \sigma^2 I)$ is stochastic and cannot be backpropagated through. The trick rewrites the sampling as $z = \mu + \sigma \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$ is a fixed noise source, moving stochasticity out of the differentiation path. This allows the whole encoder-sampling-decoder pipeline to train end-to-end via standard backpropagation.

VAEs produce a smooth, continuous latent space where interpolation yields plausible samples and random samples from the prior decode to reasonable data. However, VAEs tend to produce blurry images compared to GANs or diffusion models, because pixel-wise reconstruction loss averages over decoder uncertainty. Variants include β-VAE (adjustable KL weight for disentanglement), VQ-VAE (discrete codebook latent, used in Stable Diffusion as the image tokeniser), and hierarchical VAEs.