Long-form writing on models, tooling, and applied research.
›LeWorldModel: Stable End-to-End Joint Embedding Predictive Architecture from PixelsLeWM learns end-to-end world models from pixels using JEPA with a single SIGReg regularizer — no reconstruction loss, no pretrained features, no temporal smoothness loss. Achieves 96% success on Push-T (vs 78% PLDM, 81% DINO-WM) with ViT-S predictor. Key insight: isotropic Gaussian latent constraint (SIGReg) replaces multiple regularizers; temporal straightening emerges implicitly.Jul 19, 2026
›VL-JEPA: Joint Embedding Predictive Architecture for Vision-LanguageVL-JEPA learns a joint vision-language embedding space using only JEPA-style latent prediction + SIGReg regularization — no contrastive loss (CLIP), no generative loss (LLaVA), no reconstruction (MAE). Competitive zero-shot on ImageNet, Flickr30K/COCO retrieval, GQA/VQAv2 VQA.Jul 19, 2026
›Attention Is All You Need — Paper BreakdownThe Transformer replaces RNNs/CNNs with pure attention. Encoder-decoder stacks of self-attention + FFN achieve SOTA translation (28.4 BLEU EN-DE, 41.8 EN-FR) at fraction of training cost. Introduces multi-head attention, scaled dot-product attention, sinusoidal positional encoding.Jul 15, 2026
›Paper: DeepSeek-V3.2 — Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces **DeepSeek Sparse Attention (DSA)** for efficient long-context (128K), a **scalable GRPO RL framework** (post-training compute >10% pre-training), and **large-scale agentic task synthesis** (1800 environments, 85K prompts). V3.2-Speciale (relaxed length constraints) achieves **gold medals in IMO 2025, IOI 2025, ICPC WF 2025** and matches Gemini-3.0-Pro.Jul 15, 2026
›Paper: DeepSeek-V4 — Towards Highly Efficient Million-Token Context IntelligenceDeepSeek-V4 introduces **hybrid CSA+HCA attention** for 1M token contexts, **Manifold-Constrained Hyper-Connections (mHC)** for stable deep residual connections, and **Muon optimizer** with hybrid Newton-Schulz for faster convergence. V4-Pro (1.6T params, 49B activated) and V4-Flash (284B params, 13B activated) use FP4 experts and achieve **27% FLOPs / 10% KV cache** vs V3.2 at 1M tokens. Post-training uses independent expert cultivation + on-policy distillation.Jul 15, 2026
›FlashAttention — Paper BreakdownFlashAttention (Dao et al., ICML 2022) is an **exact** attention algorithm that avoids materializing the `N^2` attention matrix by tiling computation to fit in GPU SRAM, using online softmax for incremental normalization, and recomputing attention scores during backward pass. Achieves 2-4× speedup vs standard attention, enables 16K-32K context on same hardware.Jul 15, 2026
›Paper: Hierarchical Reasoning Model (HRM) — Deep Recursive Reasoning with Adaptive ComputationHRM introduces a **hierarchical recursive architecture** with two recurrent modules (H: slow, L: fast) that perform multi-step reasoning. Trained via **deep supervision** (multiple forward segments with detached hidden states) and **ACT (Adaptive Computation Time)** using Q-learning for dynamic halting. Achieves **40.3% ARC-AGI-1**, **87% Sudoku-Extreme** with 27M params. Key finding from TRM ablation: deep supervision is the primary driver (19%→39%), not the hierarchy itself.Jul 15, 2026
›Paper: Tiny Recursive Model (TRM) — Less is More: Recursive Reasoning with Tiny NetworksTRM (Samsung SAIL Montréal, 2025) simplifies HRM by **removing the two-module hierarchy** and using a **single tiny 2-layer network** (7M params) that recursively updates a latent state and answer. With **deep supervision** (up to 16 segments), it achieves **45% ARC-AGI-1** and **8% ARC-AGI-2** — SOTA for small models, beating HRM (40.3%/5%) and most LLMs, with <0.01% parameters. **Key finding**: deep supervision is the primary driver of HRM's gains (19%→39% ablation), not the hierarchical recursive reasoning (+3.3%).Jul 15, 2026