Summary: Inference-time scaling (test-time compute) expands compute at inference rather than training — allowing models to "think longer" on hard problems. Techniques include chain-of-thought, majority voting, tree search, adaptive computation (ACT), and recursive reasoning (HRM/TRM). This is the core of OpenAI o1-style reasoning.

The Scaling Paradigm Shift

TRAIN-TIME SCALING (GPT-3 era):  More data + more params + more train compute
                                   ↓
INFERENCE-TIME SCALING (o1 era):   Fixed model + variable inference compute per query
                                   ↓
         Easy queries → fast answer    Hard queries → extended reasoning

Techniques Taxonomy

Category Method Compute Pattern Examples
Sequential Chain-of-Thought Linear in steps CoT, Scratchpad
Voting Self-Consistency k parallel samples Majority vote, weighted
Search Tree/Beam Search Exponential (pruned) MCTS, Beam, ToT
Adaptive ACT / Early Exit Variable per input ACT, PonderNet, HRM
Recursive Flat recursion Variable depth TRM, HRM (L-module)
Verification Self-critique / reward 2-5× base Process PRM, Outcome PRM

Adaptive Computation Time (ACT) — Graves, 2016

Core idea: Learn when to stop computing via a halting unit.

Input → [Recurrent Block] → Halting Unit (p_halt) → [Loop or Exit] → Output
              ↑                    ↓
         State update         Q-learning: r = -1 per step, +R at end
  • Halting distribution: p(h_t = 1 mid x) ∈ [0,1]
  • Ponder cost: lambda sum_t p(h_t) added to loss
  • Training: Q-learning on halting decisions (reward = task success - time cost)

Used in: HRM (Q-learning halting + deep supervision), PonderNet, Universal Transformers

HRM / TRM: Recursive Inference Scaling

Model Recursion Halting Deep Supervision
HRM Hierarchical (H-module → L-module) ACT (Q-learning) Per-segment (blocks)
TRM Flat (single 2-layer net) ACT (Q-learning) Primary driver (per-step)

Key insight (TRM paper): Deep supervision = main gain; hierarchy = marginal.

OpenAI o1 / o3 Paradigm

User Query → Model generates reasoning trace (CoT) → Final answer
                  ↓
         Hidden from user (safety, IP)
                  ↓
            Process reward model (PRM) in o3
  • o1: CoT + outcome reward → RL on reasoning traces
  • o3: Process reward model → step-level verification

Compute vs Performance

Compute Budget Typical Method Performance Gain
1× Greedy / Single CoT Baseline
4-8× Self-consistency (k=8) +5-15% on reasoning
10-100× Beam search / MCTS +10-30% (diminishing)
Adaptive ACT / HRM / TRM Optimal: compute ∝ difficulty

Adaptive is optimal: Easy queries → 1×, Hard queries → 100× automatically.

In Embodied AI / Robotics

Application Method
Long-horizon planning Latent MPC (more steps = longer horizon)
Contact-rich manipulation Diffusion Policy (more denoising steps = better contact)
Sim-to-real adaptation Test-time adaptation (TTA) on real obs
Failure recovery Re-plan with more compute when stuck

Related Concepts

Sources