DeepSeek Sparse Attention (DSA)

Summary: DSA (DeepSeek Sparse Attention) replaces fixed-window sparse attention with a learned sparse pattern: a lightning indexer scores prior tokens for each query, and a token selector keeps only top-k key-value entries. Combined with MLA (Multi-head Latent Attention) in DeepSeek-V3.2/V4 and GLM-5, it reduces long-context attention cost without hard-coding locality.

Motivation

In plain English: Standard attention is O(N^2). Sliding-window attention hard-codes a local window, losing global context. DSA lets the model learn which past tokens matter via a lightweight indexer + selector, keeping global reach with sparse compute.

Architecture

Two-Stage Sparse Selection

Query token h_t  →  Lightning Indexer (learned scores I_{t,s})  →  Top-k Selector  →  Sparse Attention
                           ↓                                              ↓
                  ReLU(q^I · k^I) + weights                    Keep only high-scoring
                  Low-rank, few heads, FP8                       KV entries for attention

Lightning Indexer (Section 2.1, arxiv-2512.02556):

  • Computes index score I_{t,s} = sum_{j=1}^{H^I} w_{t,j}^I · ReLU(q^I_{t,j} · k^I_{s})
  • H^I = indexer heads (small, e.g., 8); FP8 implementation for efficiency
  • Learns q^I_{t,j}, w_{t,j}^I from query h_t; k^I_s from key h_s

Token Selector:

  • Keeps top-k KV entries by index score
  • Attention: u_t = Attn(h_t, \{c_s mid I_{t,s} ∈ Top-k(I_{t,:})\})

DSA under MLA (DeepSeek-V3.2)

From arxiv-2512.02556 Section 2.1:

  • Uses MQA mode of MLA: each latent vector (KV entry) shared across all query heads
  • Enables kernel-level sharing: each KV entry used by multiple queries
  • Open-source implementation provided for unambiguous specification

DSA Architecture Source: arxiv-2512.02556 Figure 2 — DSA instantiated under MLA (green = indexer + top-k selection)

Continued Pre-Training for DSA (arxiv-2512.02556 §2.1.1)

Stage Objective Steps Tokens LR Notes
Dense Warm-up KL-divergence: align indexer to dense attention 1,000 2.1B 10^{-3} Freeze main model, only train indexer
Sparse Training KL on selected tokens + LM loss 15,000 943.7B 7.3 × 10^{-6} Top-k=2048; detach indexer for separate optimization

Warm-up KL Loss:

L^I = sum_t D_{KL}(p_{t,:} \,middle\|\, Softmax(I_{t,:}))

where p_{t,:} = L1-normalized sum of dense attention heads over sequence dimension.

Sparse KL Loss:

L^I = sum_t D_{KL}(p_{t,S_t} \,middle\|\, Softmax(I_{t,S_t}))

where S_t = selected top-k tokens.

Evolution: DeepSeek-V4 (CSA + HCA)

From arxiv-2606.19348 §2.3, DeepSeek-V4 extends DSA into a hybrid attention:

CSA (Compressed Sparse Attention)

  • Compresses every m KV entries → 1 entry (weighted by learned compression weights)
  • Applies DSA on compressed KV (lightning indexer + top-k)
  • Adds sliding window KV for local fine-grained deps

HCA (Heavily Compressed Attention)

  • Higher compression rate m' gg m (no overlap)
  • Dense attention on heavily compressed KV
  • Also adds sliding window

Hybrid CSA + HCA: Interleaved layers achieve 27% FLOPs / 10% KV cache vs V3.2 at 1M tokens.

See csa-hca-attention for full details.

Comparison: Sparse Attention Methods

Method Pattern Complexity Global Context Learned Sparsity In V3.2/V4
DSA Indexer + Top-k O(Nk) ✅ Top-k ✅ Learned V3.2
SWA Fixed local window O(Nw) ❌ ❌ Fixed —
Longformer SWA + global tokens O(Nw) ✅ Fixed ❌ —
BigBird Random + window + global O(N) ✅ Fixed ❌ —
Routing Transformer k-means clustering O(Nsqrt(N)) ✅ Centroids ✅ —
FlashAttention Dense (IO-opt) O(N^2) ✅ Full N/A Complementary
CSA Compress + DSA O(Nk/m) ✅ Top-k ✅ V4
HCA Heavy compress + dense O(Nk/m') ✅ Compressed ✅ V4

Key Claims with Sources

Claim Source Locator Confidence
Lightning indexer computes ReLU-based similarity scores arxiv-2512.02556 §2.1, Eq. 1-2 0.95
Token selector keeps top-k KV entries for attention arxiv-2512.02556 §2.1, Eq. 3 0.95
DSA instantiated under MLA using MQA mode arxiv-2512.02556 §2.1 0.9
Dense warm-up (1K steps, 2.1B tokens) aligns indexer to dense attention arxiv-2512.02556 §2.1.1 0.95
Sparse training (15K steps, 943.7B tokens) uses detached indexer opt arxiv-2512.02556 §2.1.1 0.95
DSA learns which tokens to attend (not fixed window) web-sebastianraschka-dsa Full article 0.9
DSA + MLA in DeepSeek-V3.2 and GLM-5 web-sebastianraschka-dsa Examples section 0.9
V4 evolves DSA → CSA + HCA hybrid for 1M context arxiv-2606.19348 §2.3 0.95

Related Concepts

Sources

  • arxiv-2512.02556: DeepSeek-V3.2 paper (§2.1, §2.1.1)
  • arxiv-2606.19348: DeepSeek-V4 paper (§2.3)
  • web-sebastianraschka-dsa: Sebastian Raschka LLM Architecture Gallery explainer