Planning in Latent Space

Summary: Planning in latent space means performing MPC/zero-shot planning in the compressed latent space of a world model, rather than pixel space. This enables long-horizon planning (10-50+ steps) at low compute by operating on compact representations (e.g., 32×32×32 vs 512×512×3).

---

Why Latent Space Planning?

Factor	Pixel Space	Latent Space
State dim	512×512×3 = 786K	32×32×32 = 32K (~25× smaller)
Dynamics compute	Prohibitive	Feasible (Transformer/SSM)
Rollout length	~5-10 steps	50+ steps
Value function	Hard to learn	Easy in compact space

---

Approaches

1. Discrete MPC (DreamerV3)

• Latents: 32 categorical × 32 classes (discrete)
• Planning: Sample 25 action sequences, rollout 15 steps, evaluate via value v_psi(z)
• Execute: First action, re-plan

2. Gradient-Based Planning (iLQR, MPPI)

• Differentiate through dynamics model
• Optimize action sequence via gradients
• Requires differentiable dynamics (RSSM, SSM, Mamba)

3. Value-Guided Diffusion (2024–25)

• Diffusion policy as generator of action sequences
• Value function guides denoising toward high-return trajectories
• Long-horizon: Mamba/SSM dynamics + value-guided diffusion (>50 steps)

Diagram: (Mermaid diagram - view source for diagram code)

graph LR
    A[z_t] --> B[SSM/Mamba Dynamics]
    B --> C[Imagined Trajectory z_{t:T}]
    C --> D[Value Function v_ψ]
    D --> E[Value-Guided Diffusion]
    E --> F[Optimal Action Chunk]

---

Key Algorithms

Algorithm	Dynamics	Planner	Horizon
PlaNet (Hafner et al., 2019)	RSSM	CEM (Latent)	12
Dreamer (2020)	RSSM	Actor (Policy)	∞
DreamerV2 (2021)	RSSM (Discrete)	Actor + MPC	15
DreamerV3 (2023)	RSSM (Categorical)	Discrete MPC	15 (25 particles)
IRIS (2023)	RSSM	Implicit RL	-
Genie (2024)	Latent Action Model	Video diffusion	50+ frames
Diffusion Policy (2024)	DiT (Action diffusion)	Receding horizon	16 (chunk)
Value-Guided Diffusion (2024)	SSM/Mamba	Diffusion + value	50+

---

In Embodied AI

Task	Planner	World Model
Manipulation	Diffusion Policy (receding horizon)	— (execution only)
Navigation	DreamerV3 MPC	DreamerV3
Long-horizon manipulation	Value-guided diffusion	Mamba dynamics
Video planning	Genie latent action diffusion	Genie
Sim-to-real	Policy distillation in UniSim	UniSim

---

Related Concepts

• model-based-rl — Framework
• world-model — Learned dynamics
• dreamerv3 — SOTA latent planner
• diffusion-policy — Execution layer
• value-guided-diffusion — Long-horizon planner
• mamba-ssm — Efficient long-horizon dynamics
• embodied-ai — Application domain

---

Sources

• source-dreamerv3 — DreamerV3 planning details
• world-models-diffusion-ai-robotics — Long-horizon recommendation