LeWorldModel — Paper Breakdown

TL;DR: LeWM learns end-to-end world models from pixels using JEPA with a single SIGReg regularizer — no reconstruction loss, no pretrained features, no temporal smoothness loss. Achieves 96% success on Push-T (vs 78% PLDM, 81% DINO-WM) with ViT-S predictor. Key insight: isotropic Gaussian latent constraint (SIGReg) replaces multiple regularizers; temporal straightening emerges implicitly.

Paper Metadata

Field	Value
Paper ID	`leworldmodel-paper`
Title	LeWorldModel: Stable End-to-End Joint Embedding Predictive Architecture from Pixels
Authors	(Meta FAIR / LeCun lab — check author list)
Venue/Year	arXiv 2603.19312v3 (2026)
ArXiv/DOI	arXiv:2603.19312v3
Code	Not yet released (as of paper v3)

Problem Statement

Prior JEPA world models (PLDM, DINO-WM) require:

Pretrained visual features (DINOv2) — not end-to-end from pixels
Multiple regularization terms (VICReg-style variance/covariance + temporal smoothness)
Reconstruction decoder for visualization/probing — but hurts planning performance

LeWM asks: Can we train end-to-end from pixels with a single well-designed regularizer?

Method Overview

Architecture

Observations o_{t:t+2} → ViT Encoder → Latents z_{t:t+2}
                                    ↓
                            ViT-S Predictor (autoregressive, action-conditioned)
                                    ↓
                            Predicted z_{t+3:T} → Plan via CEM
                                    ↑
                            SIGReg(𝒩(0,I)) on each z_t

Key Components

Encoder: ViT (ViT-S default) — 3 context frames → latent
Predictor: ViT-S (best), autoregressive, action-conditioned
Decoder: Only for visualization/probing — not used in training
Regularizer: SIGReg only (single λ hyperparameter)

SIGReg Regularizer (Core Innovation)

ℒ_SIGReg = λ 𝔼ₜ[𝒟_EP(Proj(zₜ), 𝒩(0,I))]

Random projections: K vectors Rₖ ~ 𝒩(0,I), project zₜ → uₖ = Rₖ·zₜ
Epps-Pulley distance: Compare characteristic function of uₖ to 𝒩(0,1)
CF matching: ∫|φ_{uₖ}(ω) - exp(-ω²/2)|² dμ(ω) ≈ ∑_m w_m |...|² (M knots)
Result: Latents match isotropic Gaussian → variance=1, covariance=0, no temporal constraint!

Technical Deep Dive

Why SIGReg Works (Intuition)

Desired Property	Standard Approach	SIGReg Approach
Unit variance	Explicit variance term	Implicit: 𝒩(0,I) has var=1
Decoupled dims	Explicit covariance term	Implicit: 𝒩(0,I) has cov=0
Temporal smoothness	Explicit temporal loss	None — emerges!
Hyperparameters	3+ sensitive terms	1 (λ) — robust ∈ [0.01, 0.2]

Emergent temporal straightening: Without temporal loss, latent velocities become collinear over training (cosine similarity → 1). This helps planning (CEM works better on straighter paths).

Predictor Design

ViT-S optimal (Tiny: 80.7%, Small: 96.0%, Base: 86.7%)
Dropout p=0.1 critical (0.0: 78%, 0.1: 96%, 0.5: 67%)
CEM planner best (96% vs Adam 84%, RMSProp 67%, SGD 26%)

No Decoder During Training

Config	Push-T SR
LeWM w/o decoder	96.0%
LeWM + decoder loss	86.0%

Decoder hurts! Pixel recon forces encoding irrelevant visual detail.

Results Summary

Main Benchmarks (Table 1)

Environment	LeWM	PLDM	DINO-WM
TwoRoom (nav)	94%	94%	90%
Push-T (manip)	96%	78%	80.7%
OGBench-Cube (3D)	82%	84%	80%
Reacher (control)	94%	86%	88%

Probing (Tables 3, 4)

Property	LeWM vs PLDM
Position (agent, block, EE)	LeWM ≥ PLDM
Velocity / Rotation	PLDM / DINO-WM better
Block orientation (quat/yaw)	All struggle

Interpretation: LeWM learns positional info equally well; DINO-WM's pretrained features help with dynamic/rotational properties.

Violation-of-Expectation

Detects both visual (color change) and physical (teleport) perturbations
Surprise signal spikes on physical violations
More consistent than PLDM/DINO-WM across environments

Ablations (Appendix G)

Ablation	Push-T SR	Insight
Embedding dim < 184	Drops	Need sufficient capacity
Num projections (K)	Flat	Robust
Num knots (M)	Flat	Robust
λ ∈ [0.01, 0.2]	>80%	λ only sensitive hyperparam
Predictor size	ViT-S best	Capacity/optimization tradeoff
Decoder loss	-10%	Hurts planning
Encoder: ViT vs ResNet-18	ViT slightly better	Architecture agnostic
Predictor dropout p=0.1	Critical	Regularizes dynamics

Strengths

True end-to-end from pixels — no DINOv2 dependency
Single regularizer (SIGReg) — replaces 3+ terms in PLDM
Hyperparameter robustness — λ ∈ [0.01, 0.2] all work; K, M insensitive
Training stability — Low variance across seeds (96.0±2.83% vs PLDM 78.0±5.0%)
Emergent temporal straightening — No temporal loss needed; straightening correlates with performance
Decoder-free training — Reconstruction actually hurts downstream planning

Weaknesses / Limitations

Velocity/rotation probing weaker than DINO-WM (pretrained features help dynamics)
No large-scale visualization — decoder only for probing, not trained
Compute: ViT encoder + predictor from scratch = more compute than PLDM (frozen DINOv2)
Single-task per model — trained per environment (10k episodes, 10 epochs)
Action space: Continuous control only; no discrete/high-dimensional actions tested

Connections to Your Wiki

Concepts This Paper Introduces / Advances

sigreg-regularizer — Core regularization technique
temporal-straightening — Emergent latent geometry property
leworldmodel — End-to-end JEPA world model

Concepts This Paper Builds On

joint-embedding-predictive-architecture — JEPA paradigm
world-model — Latent dynamics for planning
planning-in-latent-space — CEM in latent space
model-based-rl — Planning with learned dynamics
pldm — Prior JEPA world model (Pretrained Latent Dynamics Model)
dino-wm — DINOv2-based world model
vicreg — Variance-Invariance-Covariance regularization (SIGReg replaces)

Potential Project Ideas

project-sigreg-ablation — Implement SIGReg in simpler JEPA setting
project-temporal-straightening-analysis — Measure straightening in other JEPA models
project-end-to-end-jepa-vision — Scale LeWM to larger vision datasets

Reproducibility / Implementation Notes

Aspect	Detail
Compute	Not specified, but ViT-S enc/pred from scratch > frozen DINOv2
Key Hyperparams	λ=0.09 (optimal), K=64 projections, M=16 knots, p=0.1 dropout
Training	10 epochs, 10k episodes per env, offline data
Gotchas	Decoder must be DISABLED during training; predictor dropout critical

Open Source Status

Code not yet released (paper v3)
Watch for FAIR repo release

Sources

leworldmodel-paper: Full paper — primary source for all claims above