Attention Residuals (AttnRes)

Summary: Attention Residuals (AttnRes) is a depth-scaling mechanism in Kimi K3 that selectively retrieves representations from specific layers across the network depth, rather than uniformly accumulating residuals through all layers. This enables effective training and inference at 100+ layer depths.

Overview

Standard transformer residual connections add each layer's output to the accumulated stream: x_{l+1} = x_l + Layer(x_l). At extreme depths (100+ layers), this causes representation dilution and gradient degradation. AttnRes replaces uniform accumulation with selective cross-depth retrieval — each layer can "attend" to representations from any previous layer.

Key Components

Selective Cross-Depth Retrieval

  • Mechanism: Each layer computes attention over a memory bank of previous layer representations
  • Query: Current layer's hidden state
  • Keys/Values: Stored representations from selected previous layers
  • Selection: Learned or heuristic — which layers to retrieve from

Memory Bank

  • Stores hidden states from designated "checkpoint" layers
  • Compressed/quantized for memory efficiency
  • Updated periodically during training

Integration with KDA

  • KDA handles sequence-length scaling (horizontal)
  • AttnRes handles depth scaling (vertical)
  • Together enable 3T parameter scaling

Mathematical Formulation

In plain English: Instead of x_{l+1} = x_l + F(x_l), each layer computes x_{l+1} = AttnRes(x_l, {x_{l_i}}) where {x_{l_i}} are retrieved representations from earlier layers.

Standard residual:

x_{l+1} = x_l + LayerNorm(Attention(x_l) + FFN(x_l))

AttnRes:

Retrieved = CrossDepthAttention(x_l, MemoryBank)
x_{l+1} = x_l + LayerNorm(Attention(x_l) + FFN(x_l) + Retrieved)

Where MemoryBank = {x_{l_1}, x_{l_2}, ..., x_{l_k}} from selected layers.

Variants / Extensions

  • Sparse AttnRes: Only retrieve from logarithmically spaced layers
  • Learned AttnRes: Learn which layers to retrieve via gating
  • Quantized Memory: Compress memory bank with MXFP4 for inference

Applications

  • Kimi K3 (100+ layers, 2.8T params)
  • Ultra-deep transformers (>200 layers)
  • Models with mixture-of-experts at depth
  • Recursive reasoning architectures (HRM, TRM)

Historical Context

Year Work Contribution
2017 Transformer (Vaswani) Standard residual connections
2018 DeepNet (Wang et al.) Init scaling for 1000-layer transformers
2019 ReZero / SkipInit Learnable residual scaling
2020 Highway Networks Gated residuals
2023 HRM (Recursive) Depth recurrence with shared weights
2026 AttnRes (Kimi K3) Selective cross-depth retrieval for 3T scale

Related Concepts

Sources

  • kimi-k3-tech-blog: Architecture and Infrastructure — AttnRes selectively retrieves representations across depth; works with KDA
  • kimi-k3-doc: Introduction — Attention Residuals selectively retrieve representations across depth rather than accumulating uniformly