Paper Breakdown: DeepSeek-V4
Summary: DeepSeek-V4 introduces hybrid CSA+HCA attention for 1M token contexts, Manifold-Constrained Hyper-Connections (mHC) for stable deep residual connections, and Muon optimizer with hybrid Newton-Schulz for faster convergence. V4-Pro (1.6T params, 49B activated) and V4-Flash (284B params, 13B activated) use FP4 experts and achieve 27% FLOPs / 10% KV cache vs V3.2 at 1M tokens. Post-training uses independent expert cultivation + on-policy distillation.
1. Key Contributions
| # | Contribution | Section | Impact |
|---|---|---|---|
| 1 | Hybrid CSA + HCA Attention | 2.3 | 1M context; 27% FLOPs, 10% KV cache vs V3.2 |
| 2 | mHC (Manifold Hyper-Connections) | 2.2 | Stable deep residual connections via Birkhoff polytope |
| 3 | Muon Optimizer (Hybrid Newton-Schulz) | 2.4 | Faster convergence, better stability for matrix params |
| 4 | Infrastructure: Fused MoE kernel, TileLang, FP4, heterogeneous KV cache | 3 | Training/inference efficiency |
| 5 | Post-training: Expert cultivation + on-policy distillation | 3.6 | Unified model from domain specialists |
2. Architecture Overview (Fig 2)
DeepSeek-V4 Block:
┌─────────────────────────────┐
│ mHC (Manifold HC) │ ← Input residual
├─────────────────────────────┤
│ Attention (CSA or HCA) │ ← Alternating layers
├─────────────────────────────┤
│ mHC │ ← Post-attention residual
├─────────────────────────────┤
│ DeepSeekMoE FFN │ ← Mixture of Experts
├─────────────────────────────┤
│ mHC │ ← Post-FFN residual
└─────────────────────────────┘
- Retains: DeepSeekMoE, MTP (Multi-Token Prediction)
- New: CSA/HCA hybrid, mHC, Muon
3. mHC: Manifold-Constrained Hyper-Connections (§2.2)
Standard HC Problem
- Expand residual:
R^d to R^{n_{hc} × d} X_{l+1} = B_l X_l + C_l F_l(A_l X_l)B_lunconstrained → spectral norm unbounded → instability in deep stacks
mHC Solution
Constrain B_l to Birkhoff polytope M (doubly stochastic matrices):
B_l ∈ M = \{M mid Mmathbf{1}=mathbf{1},\; mathbf{1}^T M = mathbf{1}^T,\; M ≥ 0\}
→ ||B_l||_2 ≤ 1 (non-expansive) → stable propagation
A_l, C_l constrained via Sigmoid: non-negativity + boundedness
Dynamic Parameterization:
- Input
X_l to hat{X}_l = RMSNorm(vec(X_l)) tilde{A}_l = alpha_l^pre(hat{X}_l W^pre_l) + S^pre_ltilde{B}_l = alpha_l^resMat(hat{X}_l W^res_l) + S^res_l→ Sinkhorn-Knopp →B_ltilde{C}_l = alpha_l^post(hat{X}_l W^post_l)^T + S^post_l→ Sigmoid →C_l
Sinkhorn-Knopp (20 iters):
M^{(0)} = exp(tilde{B}_l), quad M^{(t)} = T_r(T_c(M^{(t-1)}))
4. Hybrid Attention: CSA + HCA (§2.3)
CSA: Compressed Sparse Attention (§2.3.1)
KV Compression (ratio
1/m):- Two streams
C^a, C^bwith weightsZ^a, Z^b - Overlapped: each compressed block uses
2mentries - Softmax with positional biases
B^a, B^b C^Comp_i = sum S^a_j odot C^a_j + sum S^b_j odot C^b_j
- Two streams
DSA on Compressed KV (lightning indexer + top-k):
- Indexer keys compressed same way →
K^IComp - Low-rank indexer queries from shared latent
c_t^Q - Top-k selection:
C^SprsComp_t = \{C^Comp_s mid I_{t,s} ∈ Top-k\} - MQA core attention + grouped output projection
- Indexer keys compressed same way →
Sliding window KV (uncompressed) for local deps
HCA: Heavily Compressed Attention (§2.3.2)
- Single KV stream, compression ratio
1/m'(m' gg m) - No overlap, no sparse selection → dense on compressed KV
- Same MQA + grouped projection
- Sliding window for local deps
Additional Techniques (§2.3.3)
| Technique | Purpose |
|---|---|
| Query/KV RMSNorm | Prevent exploding logits |
| Partial RoPE (last 64 dims) | Relative position on core outputs |
| Sliding window | Local fine-grained deps |
| Attention sink | Learnable sink logits per head |
Efficiency (§2.3.4, Fig 1 right)
| Metric | V3.2 (128K) | V4-Pro (1M) | V4-Flash (1M) |
|---|---|---|---|
| Single-token FLOPs | 100% | 27% | 10% |
| KV Cache Size | 100% | 10% | 7% |
| KV Storage | BF16 | FP8 (non-RoPE) + BF16 (RoPE) | Same |
| Indexer Compute | FP8 | FP4 | FP4 |
| Expert Weights | BF16/FP8 | FP4 | FP4 |
vs BF16 GQA8 baseline (1M ctx): V4 KV cache ≈ 2%
5. Muon Optimizer (§2.4, Algorithm 1)
Algorithm
for each weight matrix W ∈ ℝ^{n×m}:
G_t = ∇_W L_t(W_{t-1}) # Gradient
M_t = μ M_{t-1} + G_t # Momentum
O'_t = HybridNewtonSchulz(μ M_t + G_t) # Nesterov + orthogonalization
O_t = O'_t * √max(n,m) * γ # Rescale RMS
W_t = W_{t-1} * (1 - ηλ) - η O_t # Weight decay + update
Hybrid Newton-Schulz (10 steps)
| Steps | Coefficients (a,b,c) | Purpose |
|---|---|---|
| 1-8 | (3.4445, -4.7750, 2.0315) | Rapid convergence to singular values ≈ 1 |
| 9-10 | (2, -1.5, 0.5) | Stabilize exactly at 1 |
- Pre-normalize:
M_0 = M / ||M||_F - Avoids QK-Clip (muon_kimi) because RMSNorm on queries/KV prevents exploding logits
Optimizer Assignment
| Parameter Group | Optimizer |
|---|---|
| Embeddings, prediction head, RMSNorm, mHC static biases/gating | AdamW |
| All others (attn, MoE, MTP) | Muon |
6. Model Variants & Specs
| Spec | V4-Pro | V4-Flash |
|---|---|---|
| Total Params | 1.6T | 284B |
| Activated Params | 49B | 13B |
| Context Length | 1M | 1M |
| Experts | FP4 | FP4 |
| Activated Experts | 8/256 + 2 shared | 6/64 + 1 shared |
| Layers | 60 | 30 |
| Hidden Dim | 8192 | 4096 |
| Attention Heads | 64 | 32 |
7. Training
Pre-training
- V4-Flash: 32T tokens
- V4-Pro: 33T tokens
- Native 1M context support from base model
Post-Training Pipeline (§3.6)
Independent Expert Cultivation (per domain):
- SFT on domain data (math, code, agent, instruction)
- RL (GRPO) with domain-specific reward model
- → 4+ specialized experts
On-Policy Distillation (unification):
- Unified student trained via reverse KL to teachers
- Student acts on-policy, teachers as experts
- Based on: online distillation (Lu et al. 2025), MiniLLM
Infrastructure Optimizations
| Component | Innovation |
|---|---|
| MoE Kernel | Single fused kernel: comp + comm + memory overlap |
| TileLang | DSL for kernels: productivity + efficiency |
| Deterministic Kernels | Bitwise reproducibility train/inference |
| Autograd Extension | Tensor-level checkpointing for fine-grained recomputation |
| Hybrid ZeRO + Muon | Cost-effective for orthogonalization |
| mHC Recompute | Fused kernels for Sinkhorn |
| 2-Stage Context Parallel | For compressed attention |
| FP4 QAT | MoE experts + indexer QK path |
| Heterogeneous KV Cache | On-disk storage + shared-prefix reuse |
8. Evaluation Summary (§1, Fig 1 left)
Knowledge
- V4-Pro-Max > open models on SimpleQA, Chinese-SimpleQA
- Marginal lead on MMLU-Pro, HLE, GPQA
- Gap vs Gemini-3.1-Pro narrowing
Reasoning
- V4-Pro-Max > GPT-5.2, Gemini-3.0-Pro on standard benchmarks
- Slightly below GPT-5.4, Gemini-3.1-Pro (~3-6 months behind)
- V4-Flash-Max: comparable to GPT-5.2/Gemini-3.0-Pro with larger thinking budget
Agent
- Public: ≈ Kimi-K2.6, GLM-5.1; < closed frontier
- Internal: > Claude Sonnet 4.5; ≈ Opus 4.5
Long-Context (1M)
- Strong on synthetic & real use cases
- Surpasses Gemini-3.1-Pro on academic benchmarks
9. Limitations (§5)
- World knowledge still behind proprietary (less pre-train FLOPs)
- Token efficiency: longer CoT than Gemini-3.0-Pro for same quality
- Complex reasoning inferior to frontier
10. My Research Connections
| Insight | My Wiki Link |
|---|---|
| CSA/HCA → efficient ViT attention | csa-hca-attention, vision-transformer-dsa-integration |
| mHC → stable deep residual connections | mhc-hyperconnections |
| Muon → faster LLM training | muon-optimizer |
| 1M context → long-horizon video/agent tasks | Relevant for world models |
| FP4 experts → quantization research | quantization-awareness-training |
| Expert distillation → model merging | Could apply to vision experts |
Related Wiki Pages
- csa-hca-attention — Hybrid attention detail
- mhc-hyperconnections — Residual connection detail
- muon-optimizer — Optimizer detail
- deepseek-ai — Lab profile
- paper-deepseek-v3-2 — Predecessor (DSA)
- vision-transformer-dsa-integration — My ViT+DSA project
Sources
- arxiv-2606.19348: DeepSeek-V4 paper (full)