Summary: Scaling laws describe how language model performance (loss, downstream accuracy) varies with model size (N), dataset size (D), and compute (C). The Chinchilla (Hoffmann et al., 2022) result established compute-optimal scaling: for a given compute budget, model size and data should scale proportionally. Emergent abilities appear as sharp phase transitions not predicted by smooth loss curves.

The Scaling Law Formulation

Kaplan et al. (2020) — Original Scaling Laws

L(N, D) = (A)/(N^alpha) + (B)/(D^beta) + E

Where:

  • N = non-embedding parameters
  • D = training tokens
  • L = test loss (cross-entropy)
  • A, B, E, alpha, beta = fitted constants

Key finding: alpha ≈ 0.076, beta ≈ 0.095 — model size matters slightly more than data.

Chinchilla (Hoffmann et al., 2022) — Compute-Optimal

For fixed compute C ≈ 6 N D (FLOPs):

L(C) ≈ L_∞ + (K)/(C^gamma) quad with quad gamma ≈ 0.5

Optimal allocation: N ∝ D ∝ sqrt(C)

Compute Increase Model Size (N) Data (D)
1× (baseline) 1× 1×
10× 3.16× 3.16×
100× 10× 10×
1000× 31.6× 31.6×

Visualizing Scaling Regimes

Diagram: (Mermaid diagram - view source for diagram code)

graph TD
    subgraph UNDER["Under-trained (Data-constrained)"]
        U1[Too much model, too little data]
        U2[High N, Low D]
        U3[Wasted capacity — GPT-3 regime]
    end

subgraph OPTIMAL["Compute-Optimal (Chinchilla)"]
    O1[N ≈ D scaling]
    O2[Minimal loss for given compute]
    O3[Chinchilla, PaLM-2, Llama-2, GPT-4*]
end

subgraph OVER["Over-trained (Model-constrained)"]
    V1[Too much data, too little model]
    V2[Low N, High D]
    V3[Diminishing returns on data]
end

classDef under fill:#ffebee,stroke:#f44336;
classDef opt fill:#e8f5e9,stroke:#4caf50;
classDef over fill:#fff3e0,stroke:#ff9800;
class U1,U2,U3 under;
class O1,O2,O3 opt;
class V1,V2,V3 over;

Compute-Optimal Training: Practical Rules

Diagram: (Mermaid diagram - view source for diagram code)

flowchart LR
    BUDGET[Compute Budget C FLOPs] --> CALC[Calculate Optimal N, D]
    CALC --> N[N ≈ (C/6)^{1/2} × k_N]
    CALC --> D[D ≈ (C/6)^{1/2} × k_D]
    N --> TRAIN[Train N-param model on D tokens]
    D --> TRAIN
    TRAIN --> EVAL[Evaluate Loss / Downstream]
    EVAL --> ITERATE{More Compute?}
    ITERATE -- Yes --> BUDGET
    ITERATE -- No --> DEPLOY[Deploy Model]

classDef proc fill:#e3f2fd,stroke:#90caf9;
classDef data fill:#fff3e0,stroke:#ffcc02;
class BUDGET,CALC,TRAIN,EVAL,ITERATE,DEPLOY proc;
class N,D data;

For Your Projects (Consumer GPU / Modal)

Target Compute (FLOPs) Model Size (N) Tokens (D) Est. GPU-Hours (A100)
Tiny (debug) 1e18 10M 10B 0.1 hrs
Small (CIFAR/ WikiText) 1e20 100M 100B 10 hrs
Medium (Paper-worthy) 1e22 1B 1T 1,000 hrs
Large (SOTA pursuit) 1e24 10B 10T 100,000 hrs

Beyond Loss: Downstream Scaling

Power-law transfer: Downstream task performance often follows:

Acc_task = Acc_∞ - (K_task)/(L(N,D)^Delta)

Where L(N,D) is the scaling law loss.

Emergent Abilities — Wei et al., 2022

Diagram: (Mermaid diagram - view source for diagram code)

graph LR
    SMALL[Small Models] -->|Smooth loss| MEDIUM[Medium Models]
    MEDIUM -->|Sharp transition| LARGE[Large Models]

SMALL -.-> E1[No chain-of-thought]
MEDIUM -.-> E2[Emergent: CoT, arithmetic]
LARGE -.-> E3[Emergent: multi-hop, code, theory of mind]

classDef em fill:#f3e5f5,stroke:#ce93d8;
class E1,E2,E3 em;

Key insight: Emergence is not predicted by loss scaling — it's a phase transition in capability space.

Modern Scaling Considerations (2024–2025)

1. Data Quality > Quantity

  • Filtered/weighted data outperforms raw CommonCrawl
  • Phi/Microsoft: Textbook-quality synthetic data → small models, big performance
  • Scaling law with quality: L ∝ D^{-beta} Q^gamma where Q = quality score

2. Architecture Matters

Architecture Scaling Exponent Notes
Standard Transformer gamma ≈ 0.5 Baseline
Mixture of Experts (MoE) Better FLOP-efficiency Sparsity helps
Mamba / SSM Different curve Linear scaling in context
DSA / CSA-HCA Your research! Sparse attention changes scaling

3. Inference-Time Scaling (Test-Time Compute)

  • Chain-of-thought = implicit compute scaling
  • Best-of-N / Majority voting = explicit compute scaling
  • Process Reward Models = guided search scaling

For Your Research (PhD Applications)

Scaling Law Research Directions Aligned With Your Profile

Direction Why It Fits You Feasibility
Scaling laws for sparse attention (DSA/CSA-HCA) Your DSA-ViT project + DeepSeek V4 expertise Very High — unique angle
Scaling laws for world model / diffusion policy Robotics + diffusion focus High — underexplored
Data quality scaling for embodied AI Robotics data scarcity High — practical impact
Inference-time scaling for planning HRM/TRM + value-guided diffusion Very High — your core theme
Compute-optimal training for consumer GPU Your compute constraint reality High — immediate utility

Key References for Your Papers/Applications

Paper Year Key Result
Kaplan et al. 2020 Original power-law scaling
Hoffmann et al. (Chinchilla) 2022 Compute-optimal: N ∝ D
Wei et al. 2022 Emergent abilities
Ganguli et al. 2022 Inverse scaling (bigger = worse on some tasks)
Muennighoff et al. 2023 Scaling laws for MoE
Biderman et al. (Pythia) 2023 Open scaling suite
Touvron et al. (Llama 2/3) 2023-24 Data quality scaling

Related Wiki Pages

  • llm-training — Practical training recipes
  • compute-optimal-training — Implementation details
  • data-quality — Data filtering/weighting
  • muon-optimizer — Optimizer scaling
  • flash-attention — Enables long-context scaling
  • efficient-training — Techniques for constrained compute
  • inference-time-scaling — Test-time compute scaling
  • dsa-attention — Your sparse attention research
  • csa-hca-attention — DeepSeek V4 scaling

Sources

  • kaplan-scaling-laws-2020 — Kaplan et al., "Scaling Laws for Neural Language Models" (2020)
  • chinchilla-scaling-2022 — Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022)
  • emergent-abilities-2022 — Wei et al., "Emergent Abilities of Large Language Models" (2022)
  • inverse-scaling-2022 — Ganguli et al., "Inverse Scaling Can Become U-Shaped" (2022)
  • llama3-scaling — Touvron et al., "Llama 3 Herd of Models" (2024)