Textbook of AI

An Autoencoder is a neural network trained to reconstruct its input through a bottleneck. It consists of an encoder $f_\theta$ mapping an input $\mathbf{x}$ to a lower-dimensional latent representation $\mathbf{z} = f_\theta(\mathbf{x})$, and a decoder $g_\phi$ mapping $\mathbf{z}$ back to a reconstruction $\hat{\mathbf{x}} = g_\phi(\mathbf{z})$. Training minimises a reconstruction loss, typically mean squared error or cross-entropy. Because the latent is smaller than the input, the autoencoder must learn to compress the data into its most salient features, a nonlinear generalisation of PCA.

Standard autoencoders learn useful representations but are poor generative models: their latent space is irregular, so random samples may decode to nonsense. The Variational Autoencoder (VAE) addresses this by imposing a probabilistic structure: the encoder outputs parameters of a distribution (typically Gaussian), and the decoder learns to reconstruct from samples drawn from it. The loss combines reconstruction error with a KL divergence term that regularises the latent distribution toward a standard Gaussian prior. The reparameterisation trick ($\mathbf{z} = \mu + \sigma \odot \epsilon$, $\epsilon \sim \mathcal{N}(0, I)$) enables gradients to flow through the sampling step.

Autoencoders are used for dimensionality reduction, denoising, anomaly detection (high reconstruction error signals an outlier), feature learning, and generative modelling. VQ-VAE uses a discrete codebook latent and has become an important building block in modern text-to-image systems, where Stable Diffusion runs its diffusion process in a VQ-VAE's compressed latent space to dramatically reduce compute.