Joint Embedding Predictive Architecture (JEPA)

Summary: JEPA is a self-supervised learning paradigm (proposed by Yann LeCun) where models predict latent representations of masked/future inputs rather than reconstructing raw pixels or contrasting views. It avoids the "pixel-level detail" burden of generative models and the "negative sampling" complexity of contrastive learning — focusing representation capacity on semantically relevant, predictable features.

Overview

JEPA sits between generative (reconstruct all pixels) and contrastive (push/pull views) SSL:

  • Generative (MAE, diffusion): Predict every pixel → wastes capacity on unpredictable noise (texture, high-freq)
  • Contrastive (CLIP, SimCLR): Need negative pairs → sampling complexity, false negatives
  • JEPA (I-JEPA, V-JEPA, LeWM): Predict latent targets → learns semantic abstractions; no negatives; no pixel recon

Key Components

Architecture

Input x → Encoder f_θ → Latent z = f_θ(x)
Target region → Target Encoder f_ξ (EMA) → Target latent z_target
Predictor g_φ(z_context) → Predicted target latent ẑ_target
Loss = ||ẑ_target - z_target||² (or similar)
  • Context encoder: Processes visible/unmasked portion
  • Target encoder: EMA of context encoder (stability)
  • Predictor: Lightweight (ViT-S), predicts target latents from context latents
  • Loss: MSE / smooth L1 in latent space (not pixel space)

Masking Strategy

  • I-JEPA (images): Block masking (large semantic patches)
  • V-JEPA (video): Tube masking (spatiotemporal blocks)
  • LeWM (control): Autoregressive action-conditioned prediction
  • VL-JEPA: Cross-modal masking (predict text from image, image from text)

Why Latent Prediction Works

  1. Semantic bottleneck: Latents discard unpredictable noise (stochastic textures)
  2. No negative pairs: Target comes from same image/video (EMA encoder)
  3. Scalable: Predictor is small; encoder can be large
  4. World model friendly: Action-conditioned prediction → planning

Variants / Extensions

Variant Domain Prediction Target Key Innovation
I-JEPA Images Masked patch latents Semantic-level masking
V-JEPA Video Masked tube latents Spatiotemporal tubes
V-JEPA 2 Video Future latents + understanding Frozen encoder, multi-task
LeWM Control (pixels) Future latents (action-cond) End-to-end from pixels, SIGReg
VL-JEPA Vision-Language Cross-modal latents Joint image-text latent space

Mathematical Formulation

In plain English: Instead of "reconstruct the missing pixels," JEPA says "predict the meaning of the missing region as represented by the target encoder." The target encoder's output is a semantic abstraction — so the predictor learns to reason about semantics, not textures.

# I-JEPA style
z_context = encoder(x_masked)           # [B, L_ctx, D]
z_target_gt = target_encoder(x_full)[:, mask_idx]  # [B, L_tgt, D] (detached)

z_pred = predictor(z_context)           # [B, L_tgt, D]
loss = mse(z_pred, z_target_gt.detach())

# Target encoder: EMA update
target_encoder = momentum * target_encoder + (1-momentum) * encoder

For LeWM (autoregressive):

z_t = encoder(o_t)                     # Current observation latent
z_next_pred = predictor(z_t, a_t)      # Predict next latent given action
loss = mse(z_next_pred, encoder(o_{t+1}).detach()) + λ * SIGReg(z_t)

Applications

  • Representation learning: I-JEPA, V-JEPA → strong linear probe / fine-tuning
  • Video understanding: V-JEPA 2 → action recognition, VQA
  • World models / Control: LeWM → planning in latent space (CEM)
  • Vision-Language: VL-JEPA → joint embeddings, retrieval, VQA

Historical Context

Year Work Contribution
2020 BYOL / SimSiam No negatives, predictor + EMA target
2021 MAE Masked pixel reconstruction
2022 I-JEPA (Assran et al.) Latent prediction for images
2023 V-JEPA (Bardes et al.) Latent prediction for video
2024 V-JEPA 2 Frozen encoder, multi-task, planning
2024 PLDM / DINO-WM JEPA for world models (pretrained features)
2026 LeWM / VL-JEPA End-to-end JEPA from pixels; SIGReg; Vision-Language JEPA

Related Concepts

Sources

  • leworldmodel-paper: Section 1, Section 2 — JEPAs predict latent representations; LeCun's autonomous intelligence path; avoids pixel recon and negative sampling
  • vl-jepa-paper: Abstract, Section 1 — VL-JEPA extends JEPA to vision-language with SIGReg, no generative loss