Summary: DreamerV3 (Hafner et al., 2023) is a model-based RL agent that learns a world model in latent space and performs actor-critic planning entirely in latent space. It achieves human-level Atari with fixed hyperparameters across domains — a major step toward general world models.

Key Innovations

Innovation Description
Fixed hyperparameters Same config works for Atari, Robotics, Control, ProcGen
Latent dynamics + reward/value World model predicts next latent, reward, continue, value
Representation learning CNN encoder + RSSM (Recurrent State-Space Model)
Actor-critic in latent space Policy/value operate on latents, not pixels
Symlog + two-hot rewards Stable reward prediction across scales
Diffusion decoder (V3+) Sharp image reconstruction from latents

Architecture

Observation x_t → Encoder → Latent z_t
                          ↓
Recurrent state h_t ← RSSM ← (h_{t-1}, z_{t-1}, a_{t-1})
                          ↓
                ┌─────────┼─────────┐
                ↓         ↓         ↓
           Decoder    Reward    Value/Continue
            (x̂_t)     (r̂_t)      (v̂_t, ĉ_t)
                ↓         ↓         ↓
           Reconstruction  →  Actor-Critic
           Loss                (Policy π(a|h,z))

RSSM (Recurrent State-Space Model):

  • Deterministic path: h_t = f_theta(h_{t-1}, z_{t-1}, a_{t-1}) (GRU)
  • Stochastic path: z_t ~ q_phi(z_t mid h_t, x_t) (posterior) / p_theta(z_t mid h_t) (prior)

DreamerV3 + Diffusion Decoder (2024)

Component Original DreamerV3 + Diffusion Decoder
Decoder CNN (blurry) Latent Diffusion (sharp)
Atari Score 1.44× human 1.68× human
Visual quality Blurry predictions Photorealistic
Planning quality Good Excellent (sharp futures)

Why it works: Diffusion decoder provides sharp predictions → better value estimates → better policy.

Fixed Hyperparameters (The Breakthrough)

Hyperparameter Value Used Across
Model size ~20M All domains
KL scale 1.0 Atari, Control, Robotics
Actor/critic LR 3e-5 / 3e-4 All
Batch size 512 All
Sequence length 64 All
Discount γ 0.99 All

"One config to rule them all" — no per-domain tuning.

Results

Benchmark DreamerV3 Prior SOTA
Atari 100k Human-level (1.44×) EfficientZero (1.32×)
Atari 200M 1.68× human (w/ diff decoder) —
DMLab Human-level —
ProcGen Strong generalization —
Robotics (real) Sample-efficient —

In World Model Stack

Layer DreamerV3
Representation CNN + RSSM latent space
Dynamics Recurrent latent dynamics
Planning Latent actor-critic (TD-MPC style)
Decoder CNN (V3) → Diffusion (V3+)

Complements: Genie (video), UniSim (trajectories), Diffusion Policy (control)

Related

Sources