SiTU Activation (Sigmoid Tanh Unit)
Summary: SiTU (Sigmoid Tanh Unit) is a gated activation function designed for MoE expert FFNs at extreme sparsity. It combines sigmoid gating with tanh nonlinearity to provide better activation control, mitigating dead-expert and gradient saturation issues in Stable LatentMoE at 2.8T scale.
Overview
At extreme MoE sparsity (1.78% activation in Kimi K3), most experts are inactive for any given token. Standard activations (ReLU, GeLU, SwiGLU) can suffer from dead neurons in rarely-activated experts and gradient pathologies when experts turn on/off. SiTU addresses this with a bounded, smoothly gated design.
Key Components
Formula
SiTU(x) = x * σ(x) * tanh(x)
Where σ is sigmoid. Alternative parameterized form:
SiTU(x; α, β) = x * σ(α*x) * tanh(β*x)
Properties
- Bounded output: tanh bounds output to [-1, 1], sigmoid gate further modulates
- Smooth gradients: No hard zeros (unlike ReLU), no saturation for negative x (unlike tanh alone)
- Self-gating: σ(x) acts as input-dependent gate — large positive x → gate ≈ 1; negative x → gate ≈ 0
- Identity near zero: For small x, SiTU(x) ≈ x * x * x → very small, stable
Role in Stable LatentMoE
- Expert FFN activation: Replaces GeLU/SwiGLU in expert feed-forward networks
- Dead expert mitigation: Smooth gating allows gradients to flow even when expert is rarely selected
- Quantization-friendly: Bounded output range aids MXFP4/MXFP8 quantization-aware training
Mathematical Formulation
In plain English: SiTU is like a "soft" version of x * tanh(x) that automatically turns down the volume for negative inputs via the sigmoid gate. It's always smooth, never completely zero-gradient, and naturally bounded.
Comparison with common activations:
| Activation | Formula | Bounded? | Zero-Gradient Region | Self-Gated? |
|---|---|---|---|---|
| ReLU | max(0, x) | No | x < 0 | No |
| GeLU | x * Φ(x) | No | None (smooth) | Implicit |
| SwiGLU | x * σ(x) ⊙ Wx | No | None | Yes (bilinear) |
| Tanh | tanh(x) | Yes [-1,1] | x | |
| SiTU | x * σ(x) * tanh(x) | Yes [-1,1] | None | Yes |
Variants / Extensions
- Parameterized SiTU: Learnable α, β for gate sharpness and tanh scale
- SiTU-GLU: Bilinear form
SiTU(x) ⊙ (Wx + b)combining with GLU - Quantized SiTU: Lookup-table or polynomial approximation for MXFP4 inference
Applications
- Stable LatentMoE expert FFNs (Kimi K3)
- Extreme sparsity MoE (>512 experts, <2% activation)
- Quantization-aware training (MXFP4/MXFP8)
- Ultra-deep sparse networks
Historical Context
| Year | Activation | Context |
|---|---|---|
| 2011 | ReLU | Standard for deep nets |
| 2016 | GeLU | Transformer default (BERT, GPT) |
| 2020 | Swish / SiLU | x * σ(x), smooth ReLU |
| 2022 | SwiGLU | PaLM, LLaMA — gated linear unit |
| 2023 | ReGLU / GeGLU | Variants |
| 2026 | SiTU (Kimi K3) | Bounded, self-gated, MoE-specialized |
Related Concepts
- stable-latentmoe
- quantile-balancing
- per-head-muon
- gated-mla
- kimi-k3
- activation-functions
- moe-expert-activation
- quantization-awareness-training
Sources
- kimi-k3-doc: Introduction — Sigmoid Tanh Unit (SiTU) improves activation control for expert FFNs
- kimi-k3-tech-blog: Architecture and Infrastructure — SiTU and Gated MLA improve activation control and attention selectivity; enable stable 2.8T training with Quantile Balancing and Per-Head Muon