14.13 Score-based generative models and SDEs
Song et al. (2021) 2020 unified denoising diffusion, score matching, and continuous flows under a single stochastic differential equation framework.
The forward SDE
Consider a stochastic differential equation
$$dx = f(x, t)\, dt + g(t)\, dw$$
with $w$ a Wiener process. For specific choices of drift $f$ and diffusion coefficient $g$ this recovers the DDPM forward chain (in the continuous-time limit) or other noise schedules. Two standard choices:
- Variance-preserving (VP) SDE: $f = -\frac{1}{2}\beta(t)x$, $g = \sqrt{\beta(t)}$. Continuous-time DDPM.
- Variance-exploding (VE) SDE: $f = 0$, $g = \sqrt{d[\sigma^2(t)]/dt}$. The noise variance explodes as $t \to T$.
The reverse SDE
Anderson (1982) showed that any SDE has a reverse-time companion. For our forward SDE the reverse-time process is
$$dx = [f(x, t) - g(t)^2 \nabla_x \log p_t(x)]\, dt + g(t)\, d\bar w$$
where $\bar w$ is a reverse-time Wiener process. The reverse SDE has the same marginals as the forward SDE in reverse, start from the noisy distribution, integrate backwards, and you get a sample from the data distribution.
The key term is the score $\nabla_x \log p_t(x)$. We do not have it analytically, but we can estimate it by training a neural network to match $\nabla_x \log p_t(x)$, denoising score matching, equivalent to the DDPM noise prediction up to a $t$-dependent reweighting.
The probability-flow ODE
For each reverse SDE there is a corresponding deterministic ODE with the same marginals:
$$dx = \left[f(x, t) - \frac{1}{2}g(t)^2 \nabla_x \log p_t(x)\right] dt$$
Integrating the probability-flow ODE gives deterministic sampling, equivalent to DDIM. This connects diffusion to continuous normalising flows: a diffusion model at sampling time is a CNF.
The SDE perspective unlocks tools from numerical analysis: high-order ODE solvers (Heun, DPM-Solver), adaptive step sizes, exact likelihood computation (via the instantaneous change of variables), and a single mathematical framework that subsumes most of the diffusion literature.