VL-JEPA — Paper Breakdown

TL;DR: VL-JEPA learns a joint vision-language embedding space using only JEPA-style latent prediction + SIGReg regularization — no contrastive loss (CLIP), no generative loss (LLaVA), no reconstruction (MAE). Competitive zero-shot on ImageNet, Flickr30K/COCO retrieval, GQA/VQAv2 VQA.

Paper Metadata

Field	Value
Paper ID	`vl-jepa-paper`
Title	Joint Embedding Predictive Architecture for Vision-Language
Authors	(Meta FAIR / LeCun lab — check author list)
Venue/Year	arXiv 2026
ArXiv/DOI	(To be filled)
Code	Not yet released

Problem Statement

Standard VLM training uses 3+ loss types:

Contrastive (CLIP, SigLIP): Image-text alignment via InfoNCE/sigmoid loss — needs negative sampling, O(batch²)
Generative (LLaVA, Flamingo, PaLI): Next-token prediction — hallucination risk, compute-heavy
Reconstruction (MAE-style): Pixel/token recovery — wastes capacity on unpredictable detail

VL-JEPA question: Can a single regularizer (SIGReg) + latent prediction replace all three?

Method Overview

Architecture

Image x_img → ViT Encoder → z_img ∈ ℝ^D
Text x_text → BERT Encoder → z_text ∈ ℝ^D
                              ↓
                    Joint latent space (shared D)
                              ↓
              SIGReg on z_img, z_text, [z_img; z_text]
                              ↓
              Masked prediction in latent space
              (predict masked image patches from text context + unmasked image)
              (predict masked text tokens from image context + unmasked text)

SIGReg in Multimodal Setting

ℒ_SIGReg = λ_img · 𝔼[𝒟_EP(Proj(z_img), 𝒩(0,I))]
         + λ_text · 𝔼[𝒟_EP(Proj(z_text), 𝒩(0,I))]
         + λ_cross · 𝔼[𝒟_EP(Proj([z_img; z_text]), 𝒩(0,I))]

Each modality's latents match 𝒩(0,I) independently
Concatenated joint latents also match 𝒩(0,I)
Random projections + Epps-Pulley CF distance (same as LeWM)

Prediction Task

Masked latent prediction: Like I-JEPA/V-JEPA but cross-modal
Image → Text: Predict masked text latents from image + unmasked text
Text → Image: Predict masked image patch latents from text + unmasked image
No pixel/token reconstruction — loss in latent space only

Technical Deep Dive

Loss Formulation

ℒ = ℒ_pred_img + ℒ_pred_text + ℒ_SIGReg

ℒ_pred_img = || predictor(z_text, z_img_unmasked) - z_img_masked_gt ||²
ℒ_pred_text = || predictor(z_img, z_text_unmasked) - z_text_masked_gt ||²

z_img_gt = target_img_encoder(img)  # EMA of online encoder
z_text_gt = target_text_encoder(text)

Why No Contrastive Loss Works

CLIP/SigLIP: Need negatives to avoid collapse (everything → same vector)
SIGReg: Prevents collapse by explicitly matching isotropic Gaussian
- Variance → 1 (no dimension collapse)
- Covariance → 0 (no dimension redundancy)
- No negative sampling needed
Prediction task: Aligns modalities semantically (predict text from image → must capture shared concepts)

Computational Advantages

Aspect	CLIP/SigLIP	VL-JEPA
Loss compute	O(B²) pairwise	O(B) per-sample
Memory	Full similarity matrix	Latent params only
Negatives	Careful sampling	None needed
Hyperparams	Temperature, negatives	λ_img, λ_text, λ_cross (robust)

Results Summary

Zero-Shot Classification

ImageNet: Competitive with CLIP/SigLIP (exact numbers in paper)
VTAB: Strong transfer

Image-Text Retrieval

Task	Dataset	VL-JEPA	CLIP / SigLIP
Image→Text	Flickr30K	Competitive	Strong
Text→Image	Flickr30K	Competitive	Strong
Image→Text	COCO	Competitive	Strong
Text→Image	COCO	Competitive	Strong

Visual Question Answering

Benchmark	VL-JEPA	Generative VLMs (LLaVA, etc.)
GQA	Competitive	Stronger
VQAv2	Competitive	Stronger

Key: VL-JEPA matches contrastive on retrieval/classification but doesn't generate — VQA via linear probe on joint latents, not generation.

Strengths

Simplest VLM loss: One regularizer family replaces contrastive + generative + reconstruction
No negative sampling: SIGReg prevents collapse intrinsically
Unified with world models: Same SIGReg works for LeWM (control) and VL-JEPA (alignment)
Non-generative: No hallucination, explicit uncertainty in latent space
Scalable loss: O(batch) not O(batch²) — better for large-batch training
Cross-modal SIGReg: Joint distribution constraint is novel vs independent per-modality

Weaknesses / Limitations

VQA weaker than generative VLMs: Latent probe vs token generation
No text generation: Cannot do captioning, dialogue, reasoning in text
Compute: Two encoders (ViT + BERT) + predictor from scratch
Alignment quality: Contrastive methods specialized for retrieval may still edge out
Code not released: Reproducibility pending

Connections to Your Wiki

Concepts This Paper Introduces / Advances

vl-jepa — Vision-language JEPA
SIGReg in multimodal setting (see sigreg-regularizer)

Concepts This Paper Builds On

joint-embedding-predictive-architecture — Core JEPA paradigm
sigreg-regularizer — Regularization technique
leworldmodel — Sister model (control) using same regularizer
clip / siglip — Contrastive baselines
multimodal-learning — General field

Potential Project Ideas

project-vl-jepa-reproduction — Implement VL-JEPA with SIGReg
project-sigreg-multimodal — Test SIGReg on other multimodal tasks
project-jepa-vs-contrastive — Ablation: SIGReg vs InfoNCE at scale

Reproducibility / Implementation Notes

Aspect	Detail
Encoders	ViT (image) + BERT-style (text)
Target encoders	EMA of online encoders
Predictor	Lightweight (few layers)
Key hyperparams	λ_img, λ_text, λ_cross (robust ranges expected per LeWM)
Masking	Block masking for image, span masking for text
Gotchas	No decoder; no contrastive; SIGReg on concatenated latents critical

Sources

vl-jepa-paper: Full paper — primary source

VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language