Summary: Consistency Models (Song et al., ICLR 2024) enable one-step or few-step generation by enforcing self-consistency on the probability flow ODE trajectory. Can be trained from scratch (isolation) or distilled from pre-trained diffusion models. Achieves similar quality to diffusion with 10-50× faster sampling.

Core Idea

Probability Flow ODE (Song et al., 2021):

(dx)/(dt) = -(1)/(2) beta(t) ∇_x log p_t(x)

Consistency function f_phi(x_t, t) ≈ x_0 maps any point on trajectory to endpoint.

Consistency property: f_phi(x_t, t) ≈ f_phi(x_{t+Delta}, t+Delta) for small Delta

Training Modes

1. Consistency Distillation (CD)

  • Input: Pre-trained diffusion model (EDM, LDM, etc.)
  • Loss: L(phi) = E [ d(f_phi(x_{t_{n+1}}, t_{n+1}), f_{phi^-}(x_{t_n}, t_n)) ]
  • Target network phi^- (EMA of phi)
  • Result: 1-2 step generation matching teacher

2. Consistency Training (CT) — Isolation

  • No teacher — train from scratch
  • Loss: Regression toward deterministic ODE solver (Heun)
  • Result: Quality slightly lower than CD but independent

Results (from Paper)

Model Method Steps FID (ImageNet 256) FID (LSUN Bedroom)
EDM (teacher) Diffusion 35 1.97 —
CM (CD) Distilled 1 3.55 3.19
CM (CD) Distilled 2 2.27 2.42
CM (CT) From scratch 1 4.67 3.81
LCM (Luo et al.) LDM distill 1 7.76 —

For Robotics / World Models

Application Benefit
Diffusion Policy + CM 1-step action generation → 100Hz+ control
World model rollouts Fast planning (few-step latent dynamics)
Video generation Real-time video synthesis (Genie, Sora variants)
Test-time scaling 1→4 steps = quality knob

LCM-LoRA (Luo et al., 2023): Train LoRA on frozen LDM → 1-step LDM. Easier than full CM.

Architecture Comparison

Property Diffusion (DDIM) Consistency Model (CD) Consistency Model (CT)
Steps to quality 10-50 1-2 1-4
Training Score matching Distillation from teacher ODE trajectory regression
Teacher needed No Yes (pre-trained diffusion) No
Quality (FID ImageNet 256) 1.97 (35 steps) 2.27 (2 steps) 4.67 (1 step)
Inference speed Baseline 10-50× faster 10-50× faster

Consistency Distillation Algorithm

Diagram: (Mermaid diagram - view source for diagram code)

flowchart TD
    subgraph TEACHER [Pre-trained Diffusion Teacher]
        T1[Sample x_0 ~ p_data]
        T2[Add noise: x_t ~ q(x_t | x_0)]
        T3[Denoise: x_{t-Δ} = f_teacher(x_t, t)]
    end

subgraph STUDENT [Consistency Model Student]
    S1[Sample x_0 ~ p_data]
    S2[Add noise: x_t, x_{t+Δ} ~ q]
    S3[Predict: f_φ(x_t, t), f_φ(x_{t+Δ}, t+Δ)]
end

T3 --> LOSS[Consistency Loss]
S3 --> LOSS
LOSS --> UPDATE[Update φ]
UPDATE --> EMA[EMA Target φ⁻]

Consistency Loss:

L(phi) = E_{x_0, t, n} ≤ ft[ dbig(f_phi(x_{t_{n+1}}, t_{n+1}), f_{phi^-}(x_{t_n}, t_n)big) right]

Where x_{t_{n+1}} is obtained by running one step of ODE solver from x_{t_n}.

LCM-LoRA: Practical Alternative for LDM

Aspect Full CM LCM-LoRA
Base model Must retrain or distill full model Freeze LDM, train LoRA only
Training cost High (full model) Low (LoRA rank 64-128)
Inference steps 1-4 1-4
Quality Slightly better Good enough for most LDM uses
Compatibility New architecture Works with any LDM (SDXL, SD3, etc.)

LCM-LoRA Training:

# Pseudocode
ldm = load_pretrained_ldm()
lora = LoRA(rank=64, alpha=128)
freeze(ldm)
optimizer = AdamW(lora.parameters())

for batch in dataloader:
    x_0 = batch
    t = sample_timesteps()
    x_t = add_noise(x_0, t)
    
    # Teacher: few-step DDIM from LDM
    with torch.no_grad():
        x_0_hat = ddim_sample(ldm, x_t, t, steps=4)
    
    # Student: 1-step CM via LoRA
    x_0_pred = lora(x_t, t)  # CM forward
    
    loss = MSE(x_0_pred, x_0_hat)
    loss.backward()
    optimizer.step()

Integration with Diffusion Policy

Diagram: (Mermaid diagram - view source for diagram code)

flowchart LR
    subgraph BASE [Diffusion Policy Base]
        OBS[Observation] --> ENC[Encoder]
        ACTION[Noisy Action] --> DIT[DiT Denoiser]
        ENC --> DIT
        DIT --> CLEAN[Clean Action]
    end

subgraph CM_DISTILL [Consistency Distillation]
    CLEAN --> TEACHER[10-step DDIM Teacher]
    NOISY[Noisy Action] --> STUDENT[1-step CM Student]
    TEACHER --> LOSS[Consistency Loss]
    STUDENT --> LOSS
end

subgraph DEPLOY [100Hz Deployment]
    OBS_DEP[Observation @ 100Hz] --> CM_STUDENT[1-step CM Policy]
    CM_STUDENT --> ACTION_DEP[Action @ 100Hz]
end

Key benefit: 10-step DDIM → 1-step CM = 10× control frequency (10Hz → 100Hz)

For Your Research

High-Impact Directions

  1. Consistency Policy — Distill Diffusion Policy to 1-2 steps for real-time control
  2. Latent Consistency World Model — Fast latent dynamics rollout for planning
  3. Video Consistency Models — Extend CM to video (Genie dynamics → 1-2 step rollout)
  4. Test-Time Scaling for Planning — 1-step for reactive, 4-step for deliberative
  5. CM-LoRA for Robotics — LoRA distillation of existing Diffusion Policy checkpoints

Implementation Priority

Priority Task Effort Impact
1 LCM-LoRA on Diffusion Policy checkpoint Low (few GPU hours) High (100Hz control)
2 Consistency Distillation of DiT dynamics Medium High (fast planning)
3 Video CM for Genie-style dynamics High High (real-time video world model)
4 CT from scratch for action diffusion High Medium (no teacher needed)

Related Wiki Pages

Sources

  • Primary: consistency-models — Song et al., "Consistency Models" (ICLR 2024) — arXiv:2310.01412
  • Extension: Luo et al., "Latent Consistency Models" (2023) — LCM-LoRA