Summary: Genie (Bruce et al., DeepMind, 2024) is an 11B parameter action-controllable world model trained on unlabeled Internet video (no action annotations). Learns latent actions from pixels, enabling controllable video generation and zero-shot policy learning. Key breakthrough: video-only world model learning without action annotations.

Architecture Overview

Video frames (T×H×W×3)
        ↓
Video Tokenizer (3D VQ-VAE) → Discrete latents z_{1:T} (T×16×16×8)
        ↓
Latent Action Model (ST-Transformer) → Latent actions a_{1:T-1} (discrete codes)
        ↓
Dynamics Transformer (MaskGIT-style)
        ↓
    p(z_t | z_{<t}, a_{<t}) — Masked prediction

Components:

  1. Video Tokenizer — 3D VQ-VAE, 4× spatial, 4× temporal compression
  2. Latent Action Model (LAM) — Infers action a_t from (z_t, z_{t+1})
  3. Dynamics Transformer — Predicts future latents given past + actions

Training Pipeline

Stage Data Objective
Tokenizer Video (public datasets) VQ-VAE reconstruction
LAM Tokenized video Predict action between frames
Dynamics Tokenized video + LAM actions MaskGIT masked prediction

Data: ~300K hours of Internet video (YouTube, etc.) — no action labels

Key Capabilities

1. Controllable Video Generation

  • Input: First frame + latent action sequence
  • Output: Controlled future frames
  • Actions: Discrete codes (learned, not annotated)

2. Zero-Shot Policy Learning

  • Extract latent actions from expert video
  • Train BC policy: pi(a_t | z_t) on (latent, action) pairs
  • Deploy in simulator / real

3. Interactive Environments

  • Turn any video into controllable environment
  • Human provides actions → model generates response

Key Specifications

Metric Value
Parameters 11B (LAM 1B + Dynamics 10B)
Video resolution 256×256, 16 fps
Latent grid 16×16×8 (spatial×spatial×temporal)
Action codes 512 discrete actions
Zero-shot control ✅ Frame + actions → video
BC from latent actions Matches supervised on some tasks

Research Relevance (Your Focus Areas)

Your Interest Genie Connection
Video-only world models Skip action annotation bottleneck entirely
Latent action discovery Architecture for HRM/TRM-style control from video
Large-scale video training Data strategy: 300K hrs Internet video for embodied AI
Controllable generation Planning interface via latent actions
World models Pure video-trained world model (no sim, no actions)

Limitations (Acknowledged in Paper)

Limitation Impact
256×256 resolution Low for detailed manipulation
Discrete latent actions Less expressive than continuous
No physics/constraints Pure generative, not simulator
Sim-to-real not shown Domain gap unknown

Architecture Diagram

Diagram: (Mermaid diagram - view source for diagram code)

graph TD
    A[Video Frames T×H×W×3] --> B[Video Tokenizer<br/>3D VQ-VAE]
    B --> C[Discrete Latents z_{1:T}<br/>T×16×16×8]
    C --> D[Latent Action Model<br/>ST-Transformer]
    D --> E[Latent Actions a_{1:T-1}<br/>512 discrete codes]
    C --> F[Dynamics Transformer<br/>MaskGIT-style]
    E --> F
    F --> G[p(z_t \| z_{<t}, a_{<t})<br/>Masked Prediction]
    G --> H[Future Latents]
    H --> I[Video Decoder<br/>VQ-VAE Decoder]
    I --> J[Generated Video Frames]

style A fill:#e1f5fe
style B fill:#fff3e0
style D fill:#fff3e0
style F fill:#fff3e0
style I fill:#e8f5e9
style J fill:#e8f5e9

Training Pipeline Diagram

Diagram: (Mermaid diagram - view source for diagram code)

flowchart LR
    subgraph T1 [Tokenizer Training]
        V1[Internet Video<br/>~300K hours] --> VQ[3D VQ-VAE]
        VQ --> Z[Discrete Latents]
    end

subgraph T2 [LAM Training]
    Z --> ST[ST-Transformer]
    ST --> LA[Latent Actions<br/>a_t = LAM(z_t, z_{t+1})]
end

subgraph T3 [Dynamics Training]
    Z --> DT[Dynamics Transformer<br/>MaskGIT]
    LA --> DT
    DT --> PRED[Predict masked z_t]
end

subgraph INF [Inference / Control]
    IF[Initial Frame] --> TOK[Tokenizer]
    TOK --> Z0[z_1]
    ACT[User Actions<br/>or Policy π] --> LA_INF[a_t]
    Z0 --> DT
    LA_INF --> DT
    DT --> ZPRED[z_{t+1}]
    ZPRED --> VDEC[Video Decoder]
    VDEC --> VOUT[Generated Frame]
end

Comparison with Related Work

Model Training Data Action Labels Latent Actions Control
Genie Internet video ❌ No ✅ Learned ✅ Latent actions
DreamerV3 Env interaction ✅ Yes ❌ No ✅ Policy
Diffusion Policy Demos (actions) ✅ Yes ❌ No ✅ Actions
VideoGPT Video ❌ No ❌ No ❌ No

For Your Research

Paper Ideas That Build on Genie

  1. Continuous Latent Actions — Replace discrete codes with continuous latent actions (diffusion-based LAM)
  2. Hierarchical Latent Actions — High-level (task) + low-level (motor) latent action hierarchy
  3. Video + Action Hybrid — Combine Internet video (no actions) + robot demos (with actions) for better sim-to-real
  4. Consistency Model Dynamics — Replace MaskGIT dynamics with Consistency Model for 1-2 step rollout
  5. Mamba Dynamics — Replace Transformer dynamics with Mamba for long-horizon video prediction

Implementation Priorities

  1. Reproduce VQ-VAE tokenizer — 3D VQ-VAE on video (consumer GPU feasible at 64×64)
  2. LAM on simplified data — Train action predictor on tokenized video
  3. Dynamics with Consistency Models — Fast rollout for planning
  4. Zero-shot BC from video — Extract actions from expert video, train BC

Related Wiki Pages

Sources

  • Primary: deepmind-genie — Bruce et al., "Genie: Generative Interactive Environments" (2024) — arXiv:2402.14848
  • Lab: Google DeepMind (Rocktäschel group)