Textbook of AI

A Generative Model learns to approximate the probability distribution $p(x)$ from which training data was drawn, enabling it to generate new samples that resemble—but are not copies of—the training examples. This contrasts with discriminative models, which learn the conditional distribution $p(y \mid x)$ and predict labels without modelling how the data itself is generated. Generative models can synthesise realistic images, coherent text, plausible molecular structures, and convincing audio, making them one of the most consequential categories of modern AI.

Major families of generative models include Variational Autoencoders (VAEs), which learn a probabilistic latent space via the evidence lower bound; Generative Adversarial Networks (GANs), which use a generator-discriminator minimax game; Diffusion Models, which learn to reverse a gradual noising process and are currently dominant for image generation; Normalising Flows, which build invertible transformations with exact likelihoods; Autoregressive Models like language models and PixelCNN, which factorise the joint distribution into a product of conditionals; and Energy-Based Models, which define unnormalised probabilities via an energy function.

Each family has distinct trade-offs. VAEs train stably and provide latent representations but produce blurry samples. GANs produce sharp samples but are unstable and can suffer mode collapse. Diffusion models offer stable training and excellent mode coverage but require many denoising steps. Autoregressive models excel at sequential data but are slow to generate. Normalising flows give exact likelihoods but are constrained by invertibility. The practical choice depends on the domain, the desired trade-offs between sample quality, training stability, speed, and the need for likelihood evaluation.