VL-JEPA (Vision-Language Joint Embedding Predictive Architecture)

Summary: VL-JEPA extends the JEPA paradigm to vision-language, learning a joint image-text embedding space using only SIGReg regularization — no contrastive loss (CLIP), no generative loss (LLaVA/Flamingo), no reconstruction (MAE). Achieves competitive zero-shot classification, retrieval, and VQA.

Details

Architecture

  • Image Encoder: ViT (Vision Transformer)
  • Text Encoder: BERT-style transformer
  • Joint Latent Space: Both modalities projected to shared embedding space
  • Regularization: SIGReg applied to image latents, text latents, and cross-modal joint distribution

SIGReg in Multimodal Setting

ℒ_SIGReg_total = λ_img * 𝔼[𝒟_EP(Proj(z_img), 𝒩(0,I))]
              + λ_text * 𝔼[𝒟_EP(Proj(z_text), 𝒩(0,I))]
              + λ_cross * 𝔼[𝒟_EP(Proj([z_img; z_text]), 𝒩(0,I))]
  • Each modality independently matches isotropic Gaussian
  • Cross-modal joint also matches isotropic Gaussian
  • Single regularizer family replaces contrastive + generative + reconstruction

Training Objective

  • Prediction Task: Masked latent prediction — predict masked image patches from text context and vice versa
  • No Contrastive: No InfoNCE / SigLIP pairwise similarity
  • No Generative: No token autoregression
  • No Reconstruction: No pixel/token decoding

Capabilities (Zero-Shot)

Task Benchmark VL-JEPA CLIP / SigLIP / Generative VLMs
Classification ImageNet Competitive Stronger (CLIP/SigLIP specialized)
Image→Text Retrieval Flickr30K, COCO Competitive Strong
Text→Image Retrieval Flickr30K, COCO Competitive Strong
VQA GQA, VQAv2 Competitive Generative VLMs stronger

Loss Comparison

Method Contrastive Generative Reconstruction SIGReg
CLIP ✓ ✗ ✗ ✗
SigLIP ✓ (sigmoid) ✗ ✗ ✗
LLaVA/Flamingo ✗ ✓ ✗ ✗
MAE ✗ ✗ ✓ ✗
VL-JEPA ✗ ✗ ✗ ✓

Significance

  1. Simplest VLM training: One regularizer family (SIGReg) replaces three loss categories
  2. Non-generative: Avoids hallucination risks, enables explicit uncertainty in latent space
  3. No negative sampling: Contrastive methods need carefully constructed negatives; SIGReg doesn't
  4. Unified with LeWM: Same SIGReg regularizer works for world models (LeWM) and VL alignment (VL-JEPA)
  5. Scalable: No O(batch²) contrastive matrix; SIGReg is O(batch) with random projections

Related Concepts

Sources

  • vl-jepa-paper: Abstract, Section 1, Section 3 — Extends JEPA to VL with SIGReg; no contrastive/generative/reconstruction losses
  • vl-jepa-paper: Section 4 — Competitive zero-shot on ImageNet, Flickr30K/COCO retrieval, GQA/VQAv2 VQA