Model-Based Reinforcement Learning

Summary: Model-Based Reinforcement Learning (MBRL) learns a model of environment dynamics (world model) and uses it for planning or policy optimization, achieving higher sample efficiency than model-free RL. Modern MBRL (DreamerV3, TD-MPC) operates in latent space for scalability.

---

MBRL vs Model-Free RL

Aspect	Model-Free RL	Model-Based RL
Sample efficiency	Low (millions of steps)	High (10-100× fewer samples)
Asymptotic performance	Often higher	Slightly lower (model bias)
Planning	None (policy only)	Explicit (MCTS, MPPI, latent)
Generalization	Policy overfits	World model generalizes
Sim-to-real	Hard	Easier (model adapts)

---

MBRL Taxonomy

1. Dyna-style (Sutton, 1990)

• Learn model → Generate synthetic data → Train model-free policy
• Examples: Dyna-Q, MBPO, STEVE

2. Latent Dynamics + Planning (Modern SOTA)

• DreamerV3 — Latent RSSM + actor-critic in latent space
• TD-MPC — Latent dynamics + MPC planning (MPPI)
• IRIS — Latent imagination + policy improvement

3. Video Prediction / World Models

• Genie — Video tokenizer + latent actions + dynamics
• UniSim — Trajectory diffusion on 1.6M real trajectories
• VideoGPT / MCVD — Autoregressive / diffusion video prediction

4. Policy Optimization with Model

• MBPO — Short rollouts in model + real data
• COMBO — Conservative offline MBRL

---

Modern Latent MBRL Architecture

Observation → Encoder → Latent z
                    ↓
             Dynamics Model: p(z' | z, a)
                    ↓
             ┌──────┴──────┐
             ↓             ↓
        Reward Model   Value Model
        (r̂)              (V)
             ↓             ↓
        Policy π(a|z) ← Planning (MCTS/MPPI)

Key design choices:

• Latent space — Compressed (RSSM, VQ-VAE, VAE) vs pixel
• Dynamics — Deterministic (GRU) + stochastic (z) or pure diffusion
• Planning — Actor-critic (Dreamer), MPC/MPPI (TD-MPC), MCTS (MuZero)
• Training — ELBO (reconstruction + KL) + RL losses

---

DreamerV3 vs TD-MPC

Component	DreamerV3	TD-MPC
Dynamics	RSSM (GRU + stochastic)	Ensemble MLP (deterministic)
Planning	Actor-critic (imagination)	MPPI on latent dynamics
Policy	Amortized (actor net)	Online optimization (MPPI)
Hyperparameters	Fixed across domains	Per-domain tuning
Strength	Generalization, sample efficiency	Asymptotic performance, control

---

In Embodied AI Stack

┌─────────────────────────────────────┐
│          PLANNING (MBRL)            │
│  DreamerV3 / TD-MPC / Latent MPC    │
├─────────────────────────────────────┤
│         WORLD MODEL                 │
│  Latent dynamics + reward + value   │
├─────────────────────────────────────┤
│        CONTROL                      │
│      Diffusion Policy (DiT)         │
├─────────────────────────────────────┤
│       PERCEPTION                    │
│  Vision encoder → Latent z          │
└─────────────────────────────────────┘

---

Key Challenges & Solutions

Challenge	Modern Solution
Model bias	Ensemble dynamics (TD-MPC), conservative penalties (COMBO)
Compound error	Short horizons + value bootstrap, latent space
High-dim observations	Latent encoders (CNN, VQ-VAE, VAE)
Stochastic environments	Stochastic latents (RSSM), diffusion dynamics
Compute cost	Amortized planning (Dreamer actor), latent MPC

---

Related Concepts

• world-model — Learned dynamics model
• dreamerv3 — SOTA latent MBRL
• planning-in-latent-space — MPC/MCTS in latent space
• td-mpc — Model-predictive control in latent space
• embodied-ai — Application domain
• model-free-rl — Contrast

---

Sources

• source-dreamerv3 — DreamerV3 (Hafner et al., 2023)
• world-models-diffusion-ai-robotics — MBRL in embodied AI stack