Summary: Diffusion models are generative models that learn to reverse a noise process — gradually denoising from pure noise to data. They achieve state-of-the-art generation quality across images, video, audio, and robot actions, and form the generative backbone of modern world models.
Introduction
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) frame generation as iterative denoising:
- Forward process: Gradually add noise to data
x_0 to x_T ~ N(0, I) - Reverse process: Learn
p_theta(x_{t-1} mid x_t)to denoisex_T to x_0
Unlike GANs (adversarial) or VAEs (likelihood lower bound), diffusion models optimize a simple denoising score matching objective — stable, scalable, high-fidelity.
Core Formulation
Forward Process (Fixed)
q(x_t mid x_{t-1}) = N(x_t; sqrt(1-beta_t) x_{t-1}, beta_t I)
q(x_t mid x_0) = N(x_t; sqrt(bar{alpha)_t} x_0, (1-bar{alpha}_t)I)
Reverse Process (Learned)
p_theta(x_{t-1} mid x_t) = N(x_{t-1}; mu_theta(x_t, t), Sigma_theta(x_t, t))
Training objective (simplified, Ho et al. 2020):
L = E_{t, x_0, epsilon} ≤ ft[ || epsilon - epsilon_theta(sqrt(bar{alpha)_t} x_0 + sqrt(1-bar{alpha)_t} epsilon, t) ||^2 right]
Key Variants
| Variant | Innovation | Use Case |
|---|---|---|
| DDPM (Ho et al., 2020) | Original denoising diffusion | Image generation |
| DDIM (Song et al., 2020) | Non-Markovian, faster sampling | Few-step sampling |
| Latent Diffusion (Rombach et al., 2022) | Diffuse in VAE latent space (LDM/Stable Diffusion) | Efficient high-res generation |
| Consistency Models (Song et al., 2023) | One-step / few-step distillation | Real-time generation |
| DiT (Peebles & Xie, 2023) | Transformer backbone (Diffusion Transformer) | Scalable, SOTA |
| Diffusion Policy (Chi et al., 2024) | Action diffusion for robot control | Embodied AI |
In Robotics / Embodied AI
Diffusion Policy (Chi et al., 2024) — The breakthrough for robot control:
- 98% success on contact-rich manipulation with 10–50 demos
- Multi-modal action distributions = handles "grasp left OR right" naturally
- DiT backbone (~100M params) — efficient for real-time control
- Real-world transfer with minimal sim-to-real gap
Related Concepts
- latent-diffusion — Diffusion in compressed latent space (Stable Diffusion)
- diffusion-policy — Action diffusion for robot manipulation
- consistency-models — One-step diffusion distillation
- dit-diffusion-transformer — Transformer backbone for diffusion
- world-model — World models using diffusion dynamics
- embodied-ai — Physical agents using diffusion for control
Sources
- world-models-diffusion-ai-robotics — Convergence with world models
- source-arxiv-2403.12022 — Diffusion Policy (primary)
- source-consistency-models — Consistency Models (Song et al., 2023)