Inkling Architecture

Summary: Thinking Machines Lab's 975B parameter open-weight multimodal MoE with unique architectural choices: short convolutions for local inductive bias, embedding RMSNorm, learned relative-position bias instead of RoPE, and hybrid local/global attention (55 local/11 global layers). Less sparse than Kimi K2.5 (4.2% vs 3.2% activation).

Overview

Inkling follows the large-MoE recipe (DeepSeek-V3 / Kimi style) but introduces four notable deviations that collectively form a distinct architectural signature. Released open-weight (Apache 2.0) with Tinker fine-tuning platform.

Key Components

1. Short Convolutions (Kernel-4)

  • Where: After K/V projections in attention; after attention output; after MLP output
  • Why: Cheap local token mixing + explicit short-range inductive bias
  • Analogy: Like depthwise conv in ConvNeXt but in transformer blocks
  • Cost: Negligible FLOPs vs attention; parameter-free or tiny params

2. Embedding RMSNorm

  • Where: Immediately after token embeddings, before first transformer block
  • Distinct from: Standard pre-attention RMSNorm inside each block
  • Purpose: Normalize embedding scale early; prevent early-layer gradient issues
  • Status: Present in HF config and Transformers implementation; "almost redundant" but explicitly enabled

3. Learned Relative-Position Bias (No RoPE)

  • Mechanism: Input-dependent relative position bias matrix
  • Claim: "Performs better and extrapolates better to longer sequences than RoPE"
  • Local layers: Bias within 512-token window (matches local attention)
  • Global layers: Bias applied to preceding 1024 tokens only; beyond that → content-only attention
  • Relation to NoPE: Similar intuition — global layers don't need positional info beyond local context

4. Hybrid Local/Global Attention

  • 55 local layers: Sliding window attention, 512-token window
  • 11 global layers: Full attention (with relative bias limited to 1024 tokens)
  • Pattern: Similar to Longformer / BigBird but with learned bias not RoPE

Mathematical Formulation

In plain English: Standard transformer: x -> RMSNorm -> Attn -> RMSNorm -> MLP. Inkling: x -> Embedding -> RMSNorm_emb -> [local attn (+conv) / global attn (+conv)] * 55/11 -> output. Position bias: B[i,j] = f(i-j, x) learned, not RoPE(x, i).

Short Convolution

# After Q/K/V projection
K = Conv1d(K_proj(x), kernel=4, groups=d_head)
V = Conv1d(V_proj(x), kernel=4, groups=d_head)

# After attention output
attn_out = Conv1d(attn_out, kernel=4, groups=d_model)

# After MLP output
mlp_out = Conv1d(mlp_out, kernel=4, groups=d_model)

Learned Relative-Position Bias

# For local layers (512 window)
bias_local[i,j] = W_pos[i-j] for |i-j| < 512 else 0

# For global layers (1024 bias window)
bias_global[i,j] = W_pos[i-j] for |i-j| < 1024 else 0

where W_pos is learned, can be input-conditioned.

Sparsity Comparison

Model Total Params Active Params Active % MoE Config
DeepSeek-V3 671B 37B 5.5% 256 experts, top-8
Kimi K2.5 ~1T 32B 3.2% ? experts, top-?
Inkling 975B 41B 4.2% 256 experts, top-6 + 2 shared
Kimi K3 2.8T ~50B* 1.78%* 896 experts, top-16

*K3: 16/896 experts active

Quantization & Deployment

  • BF16: 2 TB VRAM (8×B300 or 16×H200)
  • NVFP4 (W4A4): 600 GB VRAM (4×B300 SM100+)
  • NVFP4 (W4A16): 8×H200
  • Frameworks: SGLang, vLLM, TokenSpeed, Unsloth, HF Transformers

Significance

Architectural "surprises" on top of standard MoE recipe:

  1. Convolutions → explicit local bias (unusual in pure transformer MoE)
  2. Embedding RMSNorm → early normalization (rarely seen)
  3. No RoPE → learned relative bias (contrarian bet on RoPE limitations)
  4. Honest benchmarks → not SOTA-chasing, designed for fine-tuning (Tinker)

Good candidate for studying: Do these help fine-tuning stability? Extrapolation? Local feature extraction?

Related Concepts

Sources

  • inkling-model-card: Model Properties — 66-layer decoder-only, hybrid local(55)/global(11) attention, 256 experts top-6 + 2 shared, 1M context
  • inkling-raschka-blog: Architecture observations — kernel-4 convolutions, embedding RMSNorm, learned relative-position bias instead of RoPE
  • inkling-raschka-blog: Sparsity comparison — 4.2% active (41B/975B) vs Kimi K2.5 3.2% (32B/1T)