Kimi Delta Attention (KDA)

Summary: Kimi Delta Attention (KDA) is a hybrid linear attention mechanism that provides an efficient foundation for scaling attention to 1M+ context windows and 3T+ parameter models. Combined with Attention Residuals (AttnRes), it forms the architectural backbone of Kimi K3. KDA introduces novel challenges for KV caching, addressed via a vLLM contribution for KDA-aware prefix caching.

Overview

Standard quadratic attention scales poorly beyond 128K-256K tokens. Linear attention variants (kernelized, recurrent, state-space) trade expressivity for efficiency. KDA is a hybrid approach combining linear attention's efficiency with selective full-attention components to maintain quality at extreme scale.

Key Components

Hybrid Design

  • Linear Attention Core: Efficient O(L) attention via kernel feature map or recurrent state
  • Selective Full Attention: Retains standard softmax attention for critical positions (e.g., local window, global tokens)
  • Delta Mechanism: The "Delta" refers to computing attention as a difference/update to a recurrent state rather than full recomputation

Delta Formulation

Standard Attention:  O(L²)
Linear Attention:    O(L) via φ(Q)φ(K)ᵀV
KDA:                 O(L) base + O(W) full-attention windows

where W << L is the window size for full attention.

Integration with AttnRes

  • KDA provides efficient cross-token mixing
  • AttnRes provides efficient cross-depth mixing
  • Together: full spatiotemporal mixing at O(L + D) vs O(L² + D²)

Mathematical Formulation

In plain English: KDA maintains a compressed recurrent state that summarizes all past tokens. For recent tokens (local window), it computes exact attention. The "delta" is the update to the recurrent state from each new token.

Let φ be a feature map (e.g., SMREG, ReLU, ELU+1). Standard linear attention:

S_t = S_{t-1} + φ(k_t) v_tᵀ        # State update: O(d²)
o_t = φ(q_t)ᵀ S_t                  # Output: O(d²)

KDA Hybrid:

# Linear recurrent state (global context)
S_t = S_{t-1} + φ(k_t) v_tᵀ
o_t^linear = φ(q_t)ᵀ S_t

# Full attention window (local context)
o_t^local = Softmax(Q_local K_localᵀ/√d) V_local

# Combined
o_t = o_t^linear + o_t^local   (or gated combination)

KV cache for linear part: just the recurrent state S (size d×d, independent of L). KV cache for local window: standard KV for last W tokens.

Variants / Extensions

  • Multi-scale KDA: Multiple recurrent states at different timescales
  • Adaptive Window: Window size W varies by layer or position
  • Gated KDA: Learnable gate between linear and local outputs

Applications

  • Kimi K3 (2.8T params, 1M context)
  • Ultra-long context models (>512K tokens)
  • Models combining linear attention with MoE (KDA + Stable LatentMoE)
  • Disaggregated inference (Mooncake) where KV cache management is critical

KV Caching Challenges & vLLM Contribution

Standard prefix caching assumes full-attention KV structure. KDA's recurrent state S is:

  • Fixed size (d×d), not growing with context
  • Updated incrementally, not appended
  • Shared across all positions in prefix

vLLM contribution: KDA-aware prefix cache that:

  1. Stores recurrent state S at prefix boundaries
  2. Handles state merging for shared prefixes
  3. Enables >90% cache hit rate on coding workloads (Mooncake)

Historical Context

Year Work Contribution
2017 Transformer Quadratic attention
2020 Linear Attention (Katharopoulos) Kernelized O(L) attention
2020 Performer / Linear Transformers Random features / kernel approx
2021 RWKV / Linear Transformers Recurrent formulation
2022 RetNet / H3 / Hyena Hybrid recurrent-convolutional
2023 Mamba / SSM Selective state spaces
2024 Various Hybrid attention + linear
2026 KDA (Kimi K3) Production hybrid linear attention at 3T scale + vLLM caching

Related Concepts

Sources

  • kimi-k3-doc: Introduction — Kimi K3 built on KDA (hybrid linear attention) and AttnRes
  • kimi-k3-tech-blog: Architecture and Infrastructure — KDA provides efficient attention scaling foundation; AttnRes does cross-depth; together scale beyond 1T; KDA-aware prefix caching contributed to vLLM