Paper Breakdown: Tiny Recursive Model (TRM)

Summary: TRM (Samsung SAIL Montréal, 2025) simplifies HRM by removing the two-module hierarchy and using a single tiny 2-layer network (7M params) that recursively updates a latent state and answer. With deep supervision (up to 16 segments), it achieves 45% ARC-AGI-1 and 8% ARC-AGI-2 — SOTA for small models, beating HRM (40.3%/5%) and most LLMs, with <0.01% parameters. Key finding: deep supervision is the primary driver of HRM's gains (19%→39% ablation), not the hierarchical recursive reasoning (+3.3%).

1. Key Contributions

# Contribution Section Details
1 Single Tiny Recursive Network 2.1 2-layer MLP/Transformer (7M params), shared params across recursion
2 Deep Supervision as Primary Driver 1, 2.4 Ablation: deep sup 19%→39%; hierarchy only +3.3% (35.7%→39%)
3 Co-Recursive Latent + Answer 2.1 z_{new} = f(z_{old}, y, x), y_{new} = f_O(z_{new}, y_{old})
4 1-Step Gradient on Last Recursion Step 2.3 IFT + Neumann truncation; O(1) memory
5 SOTA on ARC-AGI with Tiny Model 3 45% ARC-AGI-1, 8% ARC-AGI-2, 87% Sudoku-Extreme

2. Motivation (§1–2)

HRM Ablation (Independent Finding)

Configuration ARC-AGI-1 Notes
Single forward pass (no recursion) 35.7% Baseline
+ Recursive hierarchical reasoning (HRM) 39.0% +3.3%
+ Deep supervision (single-step sup) 19% Without deep sup
+ Deep supervision (full) 39% +20% absolute

Conclusion: "Using deep supervision doubled accuracy over single-step supervision (going from 19% to 39% accuracy), while recursive hierarchical reasoning only slightly improved accuracy over a regular model with a single forward pass (going from 35.7% to 39.0% accuracy)."

TRM Hypothesis

Can we keep deep supervision + recursion but drop the hierarchy for better generalization?

3. Architecture (§2.1, Fig 1)

Components

Component Description
Input embedding f_I x to R^D
Recursive network f 2-layer MLP/Transformer (shared params)
Output head f_O Latent z to answer logits
Initial states y^0 (answer), z^0 (latent) — learned or zero

Recursive Step

Input: question x, current answer y, current latent z
For n = 1..N_recur:
    z = f(z, y, x)          # Recursive latent update
y = f_O(z, y)               # Answer update from final latent
  • N_{recur}: recursion depth per supervision step
  • N_{sup} ≤ 16: deep supervision segments

4. Training Loop (Pseudocode from Fig 2, §2.1)

def trm(x, y_true):
    z = z_init
    for step in range(N_sup):          # Deep supervision segments
        x_emb = embed(x)
        z, y_pred, _ = trm_step(z, x_emb)  # Recursive reasoning
        
        loss = CE(y_pred, y_true)
        # ACT loss (Q-learning halting) - same as HRM
        loss += ACT_halt(q, y_pred, y_true)
        _, _, q_next = trm_step(z, x_emb)  # Extra forward for ACT
        loss += ACT_continue(q_next, step == N_sup-1)
        
        z = z.detach()             # KEY: detach for deep supervision
        loss.backward()
        opt.step()
        
        if q[0] > q[1]: break      # Early stop
    return y_pred

def trm_step(z, x):
    z = z.detach()   # First step detached (no grad through previous segs)
    for i in range(N_recur - 2):   # Detached recursion steps
        z = f_net(z, y, x)         # f takes (z, y, x) per §2.1
    z = f_net(z, y, x)             # Last 2 steps WITH grad
    y = output_head(z)
    q = Q_head(z)
    return z, y, q

5. Theoretical Grounding (§2.3–2.4)

Fixed-Point Recursion

Assume (z^*, y^*) is fixed point:

z^* ≈ f(z^*, y^*, x), quad y^* ≈ f_O(z^*)

1-Step Gradient (IFT + Neumann truncation)

(∂ z^*)/(∂ theta) ≈ (∂ f)/(∂ theta), quad (∂ y^*)/(∂ theta) ≈ (∂ f_O)/(∂ z^*) · (∂ f)/(∂ theta)

Only backprop through last recursion step — O(1) memory.

Deep Supervision (Same as HRM)

  • Run N_{sup} segments
  • Each segment: forward → loss → update → detach latent → next segment
  • Provides frequent module feedback
  • Regularization effect (vs Jacobian-based DEQ regularization)

6. Results (§3, Abstract)

Benchmark HRM (27M) TRM (7M) LLMs (CoT+TTC)
ARC-AGI-1 40.3% 45% 34.5% (o3-mini), 21.2% (Claude 3.7)
ARC-AGI-2 5% 8% 4.9% (Gemini 2.5 Pro, high TTC)
Sudoku-Extreme 55% 87% 0%
Maze-Hard 30×30 75% 85% <20% (175M, 1M samples)

Efficiency: TRM uses 7M params (4× smaller than HRM, 400× smaller than 7B LLM) and 1K training samples.

7. Key Claims with Sources

Claim Source Locator Confidence
Single 2-layer network, 7M params arxiv-2510.04871 Abstract, §1 0.95
Recursive latent+answer updates, up to 16 sup steps arxiv-2510.04871 §2.1, Fig 1 0.95
1-step gradient approx on last recursion step arxiv-2510.04871 §2.3 0.95
Deep supervision with detached latent init arxiv-2510.04871 §2.4 0.95
45% ARC-AGI-1, 8% ARC-AGI-2 (SOTA small models) arxiv-2510.04871 Abstract 0.95
87% Sudoku-Extreme (HRM 55%), 85% Maze-Hard (HRM 75%) arxiv-2510.04871 §1 0.95
Independent ablation: deep supervision primary HRM driver arxiv-2510.04871 §2.4 0.9
Beats most LLMs with <0.01% params arxiv-2510.04871 Abstract 0.95

8. Open Questions

  1. Why 2 layers? Would 1 or 3 work differently?
  2. Latent dimension? Not specified in source; affects capacity
  3. Language tasks? Only tested on puzzles (ARC, Sudoku, Maze)
  4. ACT stability? Uses same Q-learning as HRM; HRM notes stability from Post-Norm+AdamW
  5. Comparison to Universal Transformer / Looping Transformers — similar recursion but with deep supervision

9. My Research Connections

Insight My Wiki Link
Deep supervision > hierarchy for ARC deep-supervision, trm-recursive-reasoning
TRM for visual reasoning tasks hrm-for-visual-reasoning
Tiny recursive nets → efficient video/agent models Relevant for world models
1-step gradient → scalable recursive depth General technique
1K samples → data-efficient reasoning Low-data regime research

Related Wiki Pages

Sources

  • arxiv-2510.04871: "Less is More: Recursive Reasoning with Tiny Networks" (Jolicoeur-Martineau, 2025)
  • arxiv-2506.21734: "Hierarchical Reasoning Model" (Wang et al., 2025) — ablation context