Moonshot AI

Summary: Chinese AI research lab (Moonshot AI) developing the Kimi series of frontier open-source LLMs, known for maintaining open-source parameter scale leadership and pioneering Kimi Delta Attention, Attention Residuals, and Stable LatentMoE architectures.

Details

Key Models

Kimi K3 (2.8T params, 2026) — KDA + AttnRes + Stable LatentMoE, 1M context
Kimi K2.7 Code — Specialized coding model
Kimi K2.6 — Previous generation
Kimi K2 — 1T parameter MoE model

Research Contributions

Kimi Delta Attention (KDA): Hybrid linear attention for efficient long-context scaling
Attention Residuals (AttnRes): Depth-selective representation retrieval
Stable LatentMoE: Extreme sparsity (16/896 experts) with Quantile Balancing
Quantile Balancing: Router-score quantile-based expert allocation
Per-Head Muon: Independent attention head optimization
Mooncake: Disaggregated inference architecture (>90% cache hit rate)

Infrastructure

Kimi API Platform: Commercial API with caching-aware pricing
Kimi Work: Desktop app for knowledge work (Windows, Apple Silicon)
Kimi Code: Terminal-based coding agent
vLLM community contribution: KDA prefix caching implementation

Significance

One of the few labs consistently pushing open-source frontier model scale. Their architectural innovations (KDA, AttnRes, Stable LatentMoE) address fundamental scaling bottlenecks: attention quadratic cost, representation dilution in deep networks, and MoE routing instability at extreme sparsity. The Mooncake disaggregated inference architecture demonstrates production-grade systems co-design.

Related Concepts

Sources

kimi-k3-tech-blog: Throughout — Moonshot AI develops the Kimi series, maintaining open-source frontier model scale leadership 9 of past 12 months
kimi-k3-doc: Introduction — Kimi API Platform provides K3, K2.7 Code, K2.6 and other LLM APIs with long context, multimodal understanding, and tool calling