Paper Breakdown: Hierarchical Reasoning Model (HRM)

Summary: HRM introduces a hierarchical recursive architecture with two recurrent modules (H: slow, L: fast) that perform multi-step reasoning. Trained via deep supervision (multiple forward segments with detached hidden states) and ACT (Adaptive Computation Time) using Q-learning for dynamic halting. Achieves 40.3% ARC-AGI-1, 87% Sudoku-Extreme with 27M params. Key finding from TRM ablation: deep supervision is the primary driver (19%→39%), not the hierarchy itself.

1. Key Contributions

# Contribution Section Details
1 Hierarchical Recursive Architecture 2.1 Two recurrent modules (H slow, L fast) with different update frequencies
2 Deep Supervision 2, Fig 4 Multiple forward segments; detach hidden state between; loss at every step
3 ACT with Q-Learning 2.4 Q-head on H-state predicts halt/continue; trained via Q-learning
4 1-Step Gradient Approximation 2.3 IFT + Neumann truncation; only backprop last step; O(1) memory
5 Strong ARC-AGI Results 3 40.3% ARC-AGI-1, 87% Sudoku-Extreme, 75% Maze-Hard

2. Architecture (§2.1, Fig 2)

Two-Module Hierarchy

Module Update Frequency Role
H (High) Slow (every T steps) Abstract reasoning, planning
L (Low) Fast (every step) Detailed computation, perception

Recursive Step

For segment m = 1..N_sup:
    For step t = 1..N×T:
        if t % T == 0:
            z_H = H_net(z_H, z_L, x)    # H updates every T steps
        z_L = L_net(z_L, z_H, x)         # L updates every step
    y = Output_head(z_H)

Dimensions

  • Total params: 27M
  • H-module: 4-layer Transformer
  • L-module: 4-layer Transformer
  • Hidden dim: 512
  • N=4 L-steps per H-step, T=4 (16 L-steps per segment)
  • N_{sup} ≤ 16 deep supervision segments

3. Deep Supervision (§2, Fig 4)

Core Loop

z_H, z_L = z_H_init, z_L_init
for step in range(N_sup):           # Up to 16 segments
    x_emb = embed(x)
    z_H, z_L, y_pred, q = hrm(z_H, z_L, x_emb)
    
    loss = CE(y_pred, y_true)       # Task loss
    loss += ACT_loss(q, y_pred, y_true)  # ACT halt + continue
    
    z_H, z_L = z_H.detach(), z_L.detach()  # KEY: detach
    loss.backward()
    optimizer.step()
    
    if q_halt > q_continue: break

Why It Works

Benefit Explanation
Frequent H-module feedback H updates only once per N × T steps; deep supervision gives gradient every segment
Regularization Like auxiliary losses; prevents overfitting to final step
Superior to Jacobian reg DEQ Jacobian regularization complex; deep supervision simpler, better empirical
Biological plausibility Periodic "learning moments" aligned with neural oscillations

Ablation Evidence (from TRM paper §2.4)

  • Single-step supervision: 35.7% → 39.0% (+3.3% from recursion)
  • Deep supervision: 19% → 39% (+20% absolute) — primary driver
  • Quote (TRM §2.4): "Using deep supervision doubled accuracy over single-step supervision (going from 19% to 39% accuracy), while recursive hierarchical reasoning only slightly improved accuracy over a regular model with a single forward pass (going from 35.7% to 39.0% accuracy)."

4. ACT: Adaptive Computation Time (§2.4)

Q-Head & Halting Policy

  • Q-head: hat{Q}^m = Sigma(theta_Q^T z_H^{mNT}) → (hat{Q}_{halt}, hat{Q}_{continue})
  • State: z^m (final segment H-state)
  • Actions: {halt, continue}
  • Rewards: halt → mathbf{1}\{hat{y}^m = y\}; continue → 0

Q-Targets

hat{G}_halt^m = mathbf{1}\{hat{y}^m = y\}

hat{G}_continue^m = begin{cases} hat{Q}_halt^{m+1} & m ≥ M_{max} \\ max(hat{Q}_halt^{m+1}, hat{Q}_continue^{m+1}) & else end{cases}

Q-Loss

L^m_{ACT} = Loss(hat{y}^m, y) + BCE(hat{Q}^m, hat{G}^m)

Stochastic M_{min}

  • With prob varepsilon: M_{min} ~ Uniform(\{2, …, M_{max}\}) — encourages longer thinking
  • With prob 1-varepsilon: M_{min} = 1

Halting Condition

Halt if: m > M_{max} OR (hat{Q}_halt > hat{Q}_continue AND m ≥ M_{min})

Inference-Time Scaling (§6, Fig 5c)

Phase M_{max} Behavior
Train Fixed (e.g., 8) Learns halting policy; stochastic M_{min} encourages exploration
Inference Can increase (e.g., 8 → 16) Model handles more steps; accuracy improves zero-shot

Task-dependent scaling:

  • Sudoku: Strong scaling (needs backtracking depth)
  • ARC: Minimal (few transformations needed)

5. 1-Step Gradient Approximation (§2.3)

Fixed-Point View

Assume (z_H^*, z_L^*) is fixed point of recursive dynamics:

z_H^* ≈ H(z_H^*, z_L^*, x), quad z_L^* ≈ L(z_H^*, z_L^*, x)

Gradient via IFT + Neumann Truncation

(∂ z^*)/(∂ theta) ≈ ( (∂ F)/(∂ theta) ) bigg|_{last step}

Only backprop through last forward step of the segment — O(1) memory regardless of recursion depth.

6. Training Details

Setting Value
Optimizer AdamW
LR 3 × 10^{-4}
Weight Decay 0.1
Batch Size 64
N_{sup} (train) 8
M_{max} (train) 8
varepsilon (stochastic M_{min}) 0.5
Post-Norm RMSNorm (critical for ACT stability)

Stability (Section 6)

Deep Q-learning typically needs replay buffers + target networks. HRM achieves stability via:

  1. Post-Norm architecture (RMSNorm) — bounded activations
  2. AdamW — L_∞ constrained optimization (||theta||_∞ ≤ 1/lambda)
  3. Bounded params + weight decay + post-norm → Q-learning convergence (Gallici et al. 2025)

7. Evaluation Results (§3, Tables)

Benchmark HRM (27M) TRM (7M) LLMs (CoT + TTC)
ARC-AGI-1 40.3% 45% 34.5% (o3-mini), 21.2% (Claude 3.7)
ARC-AGI-2 5% 8% 4.9% (Gemini 2.5 Pro, high TTC)
Sudoku-Extreme 55% 87% 0%
Maze-Hard 30×30 75% 85% <20% (175M, 1M samples)

Efficiency: HRM uses 27M params; TRM uses 7M (4× smaller) with better results.

8. Limitations (§5)

  1. ACT instability risk: Q-learning without replay buffers relies on architecture choices
  2. Sudoku/ARC gap: Still fails on hardest puzzles requiring deep search
  3. Language tasks: Only tested on puzzle domains (ARC, Sudoku, Maze)
  4. Hierarchy marginal value: TRM shows deep supervision > hierarchy for these tasks

9. My Research Connections

Insight My Wiki Link
Deep supervision → primary driver, not hierarchy deep-supervision, trm-recursive-reasoning
ACT Q-learning → inference-time scaling act-adaptive-computation, inference-time-scaling
HRM for visual reasoning hrm-for-visual-reasoning
1-step gradient → memory-efficient deep recursion Relevant for video world models
Post-Norm + AdamW stability → apply to other Q-learning setups General technique

Related Wiki Pages

Sources

  • arxiv-2506.21734: Hierarchical Reasoning Model paper (full)