Summary: Genie (Bruce et al., DeepMind, 2024) is an 11B parameter video world model trained on unlabeled Internet video (no action labels). It learns latent actions from pixels, enabling controllable video generation and zero-shot policy learning — a key breakthrough for learning from passive observation.

Key Innovations

Innovation Description
Unlabeled video training No action annotations needed — learns from raw YouTube/Internet video
Latent action model Discovers controllable factors of variation in video
Video tokenizer VQ-VAE style discrete latent space (like TATS)
Latent action discovery ST-Transformer + action encoder → discrete action codes
Controllable generation Condition on latent action → steer future frames

Architecture

Video frames → Video Tokenizer (VQ-VAE) → Discrete latents z_{1:T}
                                    ↓
Latent Action Model (ST-Transformer) → Latent actions a_{1:T-1}
                                    ↓
Dynamics Transformer (conditional) → p(z_t | z_{<t}, a_{<t})

Key components:

  1. Video Tokenizer — 3D VQ-VAE, 256×256 → 16×16×8 latent grid
  2. Latent Action Model — Infers action a_t transitioning z_t to z_{t+1}
  3. Dynamics Transformer — MaskGIT-style, predicts future latents given past + actions

Capabilities

Capability Description
Controllable video gen Prompt with frame + latent action sequence → controllable future
Zero-shot policy learning Extract latent actions from expert video → train policy on (z, a)
Interactive environments Turn any video into a playable environment
Action-free learning No robot/human action labels required

Results (from paper)

Metric Result
Params 11B
Training data Unlabeled Internet video (no actions)
Action labels needed Zero
Zero-shot control ✅ Frame + action → controllable video
Policy learning Latent actions → BC policy → sim-to-real

In World Model Stack

Layer Model Role
Video generation Genie Controllable video from pixels
Latent action discovery Genie No action labels needed
Policy learning BC on latent actions Sim-to-real
Planning Latent MPC In Genie's latent space

Complements: Diffusion Policy (low-level control), DreamerV3 (latent planning), UniSim (trajectory diversity)

Related

Sources