SIGReg (Sketched-Isotropic-Gaussian Regularizer)
Summary: SIGReg is a single-scalar regularization loss that enforces latent embeddings to follow an isotropic Gaussian distribution at each timestep. It uses random projections (sketching) combined with the Epps-Pulley statistic (characteristic function distance) — replacing PLDM's multiple regularization terms (variance, covariance, temporal smoothness) with one robust, hyperparameter-light objective.
Overview
Prior JEPA world models (PLDM, DINO-WM) required multiple regularization terms: variance/covariance (VICReg-style), temporal smoothness, and sometimes contrastive terms. SIGReg observes that an isotropic Gaussian latent distribution naturally encourages:
- Variance → each dimension active (diagonal covariance ≈ I)
- Covariance → dimensions independent (off-diagonal ≈ 0)
- Temporal structure → left unconstrained (emerges implicitly)
The "Sketch" part uses random projections to estimate the high-dimensional Gaussian matching efficiently.
Key Components
1. Isotropic Gaussian Target
At each timestep t, latent embeddings z_t ∈ ℝ^D should follow 𝒩(0, I).
2. Epps-Pulley Distance (Characteristic Function)
Measures distance between distributions via characteristic functions:
𝒟_EP(P, Q) = ∫ |φ_P(ω) - φ_Q(ω)|² dμ(ω)
where φ_P(ω) = 𝔼_{x~P}[exp(i ω·x)] is the characteristic function.
For Q = 𝒩(0, I), φ_Q(ω) = exp(-||ω||²/2).
3. Random Projection Sketching
High-D characteristic function integral is intractable. Sketch:
- Draw
Krandom projection vectorsR_k ∈ ℝ^Dfrom𝒩(0, I) - Project latents:
u_k = R_k·z ∈ ℝ - Compute 1D Epps-Pulley on projected scalars
𝒟_EP(Proj(z), 𝒩(0,I)) ≈ (1/K) ∑_k 𝒟_EP_1D(R_k·z, 𝒩(0,1))
1D characteristic function for 𝒩(0,1): φ(ω) = exp(-ω²/2).
4. Numerical Integration (Knots)
Approximate integral over ω with M quadrature points (knots):
𝒟_EP_1D(P, Q) ≈ ∑_{m=1}^M w_m |φ_P(ω_m) - exp(-ω_m²/2)|²
Paper finds K (projections) and M (knots) are insensitive hyperparameters — robust across wide ranges.
Mathematical Formulation
In plain English: SIGReg asks: "Do the random 1D projections of my latents look like standard Gaussians?" If yes, the full D-dimensional distribution is close to
𝒩(0,I). This is checked by comparing characteristic functions (Fourier transforms of density) at a few points.
Total loss:
ℒ = ℒ_pred + λ * ℒ_SIGReg
ℒ_SIGReg = λ * 𝔼_t[ (1/K) ∑_{k=1}^K ∑_{m=1}^M w_m | φ_{R_k·z_t}(ω_m) - exp(-ω_m²/2) |² ]
Where:
φ_{R_k·z_t}(ω_m) = (1/B) ∑_{b=1}^B exp(i ω_m (R_k·z_{t,b}))(empirical CF)K= number of projections (default ~64, robust)M= number of knots (default ~16, robust)λ= only effective hyperparameter (robust in [0.01, 0.2])
Variants / Extensions
- Analytic SIGReg: If using moment matching instead of CF (but CF captures all moments)
- Conditional SIGReg: Match
𝒩(0, I)per action? (Paper: per-timestep only) - Learned Projections: Instead of random
R, learn discriminative projections
Ablation Results (LeWM Paper)
| Component | Push-T SR ↑ |
|---|---|
| PLDM (multi-term) | 78% |
| LeWM + SIGReg (λ=0.09) | 96% |
| No SIGReg (pred only) | ~78% (collapse) |
| SIGReg λ ∈ [0.01, 0.2] | >80% (robust) |
| Vary K (projections) | Flat |
| Vary M (knots) | Flat |
Comparison to PLDM Regularizers
| Regularizer | PLDM | LeWM (SIGReg) |
|---|---|---|
| Variance (diag cov) | Explicit term | Implicit in isotropic |
| Covariance (off-diag) | Explicit term | Implicit in isotropic |
| Temporal smoothness | Explicit term | None (emerges) |
| Hyperparameters | 3+ | 1 (λ) |
| Precision sensitivity | High | Low |
Applications
- LeWorldModel (end-to-end JEPA from pixels)
- VL-JEPA (vision-language JEPA)
- Any latent-space SSL where isotropic Gaussian is a good prior
- Replacement for VICReg/SimSiam-style regularizers in JEPA
Historical Context
| Year | Work | Regularization Approach |
|---|---|---|
| 2021 | VICReg | Variance + Covariance + Invariance |
| 2022 | Barlow Twins | Cross-correlation matrix → I |
| 2023 | I-JEPA / V-JEPA | Masked prediction + teacher EMA (no explicit reg) |
| 2024 | PLDM (LeCun et al.) | VICReg-like + temporal smoothness |
| 2024 | DINO-WM | DINOv2 features + PLDM-style |
| 2026 | LeWM (SIGReg) | Single sketched CF matching to 𝒩(0,I) |
Related Concepts
- leworldmodel
- joint-embedding-predictive-architecture
- vl-jepa
- world-model
- pldm
- dino-wm
- isotropic-gaussian-prior
- characteristic-function
- random-projection
- epps-pulley-statistic
Sources
- leworldmodel-paper: Section 3.1, Appendix C.1 — SIGReg enforces isotropic Gaussian via sketched Epps-Pulley distance; single λ replaces multiple PLDM regularizers
- leworldmodel-paper: Section 3.1, Equations 3-4 — ℒ_SIGReg = λ 𝔼_t[𝒟_EP(Proj(z_t), 𝒩(0,I))]; random projections + CF matching