CSA + HCA Hybrid Attention (DeepSeek-V4)

Summary: DeepSeek-V4 introduces a hybrid attention architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA compresses KV caches then applies sparse selection; HCA uses heavier compression with dense attention. Interleaved layers achieve 1M-token context with only 27% FLOPs and 10% KV cache vs DeepSeek-V3.2.

Motivation

In plain English: Even with DSA, attention remains the bottleneck at ultra-long contexts (1M tokens). DeepSeek-V4 further compresses KV caches before attention: CSA does moderate compression + sparse selection, HCA does aggressive compression + dense attention. The hybrid layers alternate, giving both efficiency and quality.

Architecture Overview

From arxiv-2606.19348 Figure 2:

DeepSeek-V4 Layer Stack:
┌─────────────────────┐
│  mHC (Manifold HC)  │  ← Strengthened residual connections
├─────────────────────┤
│  Attention Layer    │  ← Alternating CSA ↔ HCA
├─────────────────────┤
│  DeepSeekMoE FFN    │  ← Mixture of Experts
├─────────────────────┤
│  MTP Modules        │  ← Multi-Token Prediction
└─────────────────────┘

CSA: Compressed Sparse Attention (§2.3.1)

Two-Stage Pipeline

Input H (n×d) → KV Compression (×1/m) → DSA on Compressed KV → Output
     ↓              ↓                          ↓
  Original     C^Comp (n/m × c)         Top-k Compressed KV
  Sequence     + Sliding Window               + Sliding Window

Step 1: KV Compression (Eq. 160-191)

For each m tokens, produce one compressed KV entry:

  1. Generate KV pairs & weights: Two parallel streams (C^a, C^b and Z^a, Z^b)
C^a = H W^{aKV}, quad C^b = H W^{bKV}

Z^a = H W^{aZ}, quad Z^b = H W^{bZ}
  1. Softmax-weighted compression with positional biases B^a, B^b ∈ R^{m × c}:
S = Softmax_{row}([Z^a + B^a; Z^b + B^b])

C^Comp_i = sum_j S^a_j odot C^a_j + sum_j S^b_j odot C^b_j
  • Overlapped compression: C^b for C^Comp_i overlaps with C^a for C^Comp_{i-1}
  • Effective compression: sequence length → 1/m times

Step 2: DSA on Compressed KV (Eq. 193-249)

  • Lightning indexer on compressed KV: Compress indexer keys same way → K^IComp
  • Low-rank indexer queries: h_t to c_t^Q to q^I_{t,h} (shared latent c_t^Q)
  • Top-k selection: C^SprsComp_t = \{C^Comp_s mid I_{t,s} ∈ Top-k(I_{t,:})\}
  • MQA Core Attention: Each compressed KV serves as both key and value
  • Grouped Output Projection: Split heads into g groups → project each → final output

Sliding Window Addition

  • Each query also attends to n_win uncompressed recent KV entries
  • Preserves local fine-grained dependencies within compressed block

HCA: Heavily Compressed Attention (§2.3.2)

Compression (Eq. 267-299)

  • Single KV stream: C = H W^{KV}, Z = H W^Z
  • Non-overlapping compression every m' tokens (m' gg m):
S = Softmax_{row}(Z + B)

C^Comp_i = sum_j S_j odot C_j
  • Compression ratio: 1/m' (more aggressive than CSA)

Attention (Eq. 303-326)

  • Same MQA + grouped output projection as CSA
  • No sparse selection — dense attention on heavily compressed KV
  • Also adds sliding window KV for local deps

Hybrid Configuration

From §2.3.4 and Figure 1 (right):

Metric DeepSeek-V3.2 DeepSeek-V4-Pro DeepSeek-V4-Flash
Context 128K 1M 1M
Single-token FLOPs (1M ctx) baseline (100%) 27% 10%
KV Cache Size (1M ctx) baseline (100%) 10% 7%
KV Precision BF16 FP8 (non-RoPE) + BF16 (RoPE) Same
Indexer Precision FP8 FP4 FP4
Expert Weights BF16/FP8 FP4 FP4

Key Efficiency Techniques:

  1. Mixed KV storage: BF16 for RoPE dims, FP8 for rest → ~2× cache reduction
  2. FP4 indexer computation: Accelerates attention at long context
  3. Smaller top-k than V3.2: Better short/medium efficiency
  4. Compressed attention: Fundamental FLOPs + cache reduction

Additional Techniques (§2.3.3)

Technique Purpose
Query/KV RMSNorm Before core attention; prevents exploding logits
Partial RoPE Apply RoPE to last 64 dims of queries, KV, and core outputs (with -i for outputs)
Sliding Window n_win uncompressed KV for local deps
Attention Sink Learnable sink logits z'_h added to softmax denominator; allows heads to attend to ~0 total

Comparison: CSA vs HCA

Property CSA HCA
Compression rate m (moderate) m' (heavy, m' gg m)
Overlap Yes (overlapped) No
Sparse selection Yes (DSA top-k) No (dense on compressed)
Use case Balance quality/efficiency Maximum compression
In hybrid Odd layers Even layers (alternating)

Key Claims with Sources

Claim Source Locator Confidence
Hybrid CSA+HCA for 1M token context arxiv-2606.19348 Abstract, §1, §2 0.95
CSA: compress m→1 then DSA with lightning indexer arxiv-2606.19348 §2.3.1 0.95
HCA: compress m'→1 (m'≫m) then dense attention arxiv-2606.19348 §2.3.2 0.95
Overlapped compression in CSA (2m entries per compressed) arxiv-2606.19348 §2.3.1, Eq. 177-189 0.9
V4-Pro: 27% FLOPs, 10% KV cache vs V3.2 at 1M tokens arxiv-2606.19348 Abstract, §1, Fig 1 0.95
V4-Flash: 10% FLOPs, 7% KV cache vs V3.2 at 1M tokens arxiv-2606.19348 Abstract, §1, Fig 1 0.95
FP4 for experts + indexer QK path arxiv-2606.19348 Abstract, §2.3.4 0.9
Mixed KV storage: BF16 (RoPE) + FP8 (rest) arxiv-2606.19348 §2.3.4 0.9
Sliding window + attention sink in both arxiv-2606.19348 §2.3.3 0.9

Related Concepts

Sources

  • arxiv-2606.19348: DeepSeek-V4 paper (§2.3, §2.3.1, §2.3.2, §2.3.3, §2.3.4)