Summary: Latent Diffusion Models (Rombach et al., 2022 / Stable Diffusion) run diffusion in compressed latent space of a pretrained autoencoder (4-8× spatial compression), reducing compute 3-4× vs pixel-space diffusion while maintaining quality.

Architecture

Pixel x → [Encoder] → Latent z → [Diffusion Model] → Latent z' → [Decoder] → Pixel x'
              ↑                              ↓
        Perceptual loss (LPIPS)           Denoising UNet/DiT

Autoencoder (VQ-VAE / VAE):

  • Encoder: x mapsto z ∈ R^{H/f × W/f × C} (f=4 or 8)
  • Decoder: z mapsto hat{x} with perceptual + pixel loss
  • Frozen during diffusion training

Diffusion in latent space:

  • Forward: z_t = sqrt(bar{alpha)_t} z_0 + sqrt(1-bar{alpha)_t} epsilon
  • Denoiser predicts epsilon_theta(z_t, t, c)
  • Conditioning c via cross-attention (text, class, image)

Why Latent Space?

Factor Pixel Space Latent Space
Resolution 512×512 = 262K pixels 64×64 = 4K latents (f=8)
Diffusion steps 1000 1000 (same)
Per-step compute High 3-4× lower
Memory High Low
Quality Best Near-pixel (perceptual loss)

In World Models / Robotics

Component Role
Observation encoder Pretrained VAE (frozen)
Latent dynamics Diffusion Transformer / SSM predicts z_{t+1} mid z_t, a_t
Latent decoder Frozen VAE decoder → pixel for visualization
Value guidance In latent space (DreamerV3 + Diffusion Decoder)

DreamerV3 + Diffusion Decoder achieves 1.68× human Atari vs 1.44× — sharp predictions + sample-efficient planning.

Consistency Distillation → LCM

Latent Consistency Models (Luo et al., 2023): Distill LDM into 1-4 step consistency model.

  • LCM-LoRA: Train LoRA on frozen LDM → 1-step inference
  • Used in real-time video, robotics rollouts

Related Concepts

Sources