Kimi K3

Summary: Moonshot AI's 2.8T parameter open-weight flagship model — first in the 3-trillion class — built on Kimi Delta Attention (KDA), Attention Residuals (AttnRes), and Stable LatentMoE (16/896 experts), with 1M context window and native multimodality.

Details

Architecture

• Total Parameters: 2.8 trillion
• Attention: Kimi Delta Attention (KDA) — hybrid linear attention mechanism
• Depth Residuals: Attention Residuals (AttnRes) — selective cross-depth representation retrieval
• MoE: Stable LatentMoE — 896 experts, 16 active per token (1.78% activation sparsity)
• Expert Balancing: Quantile Balancing — router-score quantile-derived allocation, no heuristic hyperparameters
• Optimizer: Per-Head Muon — independent attention head optimization
• Activation: Sigmoid Tanh Unit (SiTU) — improved activation control
• Attention Enhancement: Gated Multi-head Latent Attention (Gated MLA) — attention selectivity
• Context Window: 1M tokens (native)
• Multimodality: Native vision (images, video) + text

Training

• Quantization-Aware Training: MXFP4 weights + MXFP8 activations from SFT stage onward
• Expert Parallelism: Fully balanced, static shapes, no host sync on critical path
• Deployment Target: 64+ accelerator supernodes (high-bandwidth domain)
• Caching: KDA-aware prefix caching contributed to vLLM

Infrastructure Innovations (from Tech Blog)

• Mooncake: Disaggregated inference architecture enabling >90% cache hit rate on coding workloads
• MiniTriton: From-scratch Triton-like GPU compiler with tile-level IR over MLIR, PTX codegen — matches/exceeds Triton on roofline benchmarks, sustains nanoGPT training
• Agentic Capabilities: 48-hour autonomous chip design (nanoGPT-serving ASIC, 4mm², 1.46M cells, 0.277MB SRAM, INT4 MAC with fused dequant, 8.7k tok/s); 2-hour computational astrophysics reproduction (I-Love-Q, 3000+ LOC, interactive dashboard)

Benchmarks (Tech Blog, max reasoning effort)

Benchmark	Kimi K3	Notable Comparisons
SWE-Bench Verified (DeepSWE v1.1)	67.3%	GLM-5.2, DeepSWE leaderboard
Terminal Bench 2.1 (H20)	67.3%	vs Opus 4.8, GPT-5.6 Sol, GLM-5.2
DeepSWE v1.1 (mini-SWE-agent)	67.3%
PostTrainBench (Harbor, H20)	Competitive	vs GPT-5.5, GPT-5.6 Sol, GLM-5.2, Opus 4.8, Fable 5
Internal Knowledge Work	Consistent gains	Production-oriented workflows
MMMU-Pro (multimodal)	Competitive	vs Fable 5, Opus 4.8, GPT-5.6 Sol
OfficeQA Pro (500 tasks, PDF-as-images)	90.4 (1M ctx, no compaction)	vs Fable 5, Opus 4.8, GPT-5.6 Sol

API & Availability

• Weights Release: July 27, 2026 (announced)
• API: platform.kimi.ai, model kimi-k3
• Pricing: 0.30/MTok cache-hit input,3.00/MTok cache-miss input, $15.00/MTok output
• Cache Hit Rate: >90% on coding (Mooncake disaggregated inference)
• Clients: Kimi App (iOS/Android/HarmonyOS), Kimi Work (Win/Mac), Kimi Code (CLI), Kimi API Platform

Limitations (Acknowledged)

1. Preserved thinking history mode required — harness must pass full reasoning content; mid-session model switch degrades quality
2. Long-horizon task bias — may over-improvise on minor/ambiguous tasks; needs explicit constraints in system prompt/AGENTS.md
3. UX gap vs. Claude Fable 5 and GPT 5.6 Sol acknowledged

Significance

First open-weight model at 3T scale. Architectural innovations (KDA, AttnRes, Stable LatentMoE, Quantile Balancing, Per-Head Muon, SiTU, Gated MLA) target the fundamental scaling bottlenecks: attention efficiency, depth scaling, extreme MoE sparsity stability, and quantization-aware training. Infrastructure co-design (Mooncake, MiniTriton, custom kernels, vLLM contributions) demonstrates full-stack capability. Agentic evaluations (compiler, chip design, research reproduction) go beyond standard benchmarks.

Related Concepts

• kimi-delta-attention
• attention-residuals
• stable-latentmoe
• quantile-balancing
• per-head-muon
• situ-activation
• gated-mla
• mooncake-inference
• minitriton
• kimi-k2

Sources

• kimi-k3-doc: Introduction — 2.8T parameters, KDA + AttnRes + Stable LatentMoE (16/896), 1M context, native vision
• kimi-k3-doc: Architecture — Stable LatentMoE with Quantile Balancing, Per-Head Muon, SiTU, Gated MLA
• kimi-k3-tech-blog: Architecture and Infrastructure — KDA efficient attention foundation; AttnRes selective cross-depth retrieval; MXFP4/MXFP8 QAT from SFT
• kimi-k3-tech-blog: Benchmarks — SWE-Bench 67.3%, Terminal Bench 67.3%, PostTrainBench competitive, OfficeQA 90.4 (1M ctx)
• kimi-k3-tech-blog: Availability — Weights by July 27 2026; API 0.30/3.00/$15.00 per MTok; >90% cache hit via Mooncake
• kimi-k3-tech-blog: Agentic Capabilities — MiniTriton compiler, 48hr chip design, 2hr astrophysics reproduction