Summary: Fixed-point recursion models (Deep Equilibrium Models, HRM, TRM) frame reasoning as finding a fixed point z^* = f_theta(z^*, x) rather than unrolling a fixed-depth network. This enables adaptive depth (inference-time scaling) and memory-efficient training via implicit differentiation.

Fixed-Point View of Recursion

Explicit Unrolling (Standard RNN)

z_{t+1} = f_theta(z_t, x), quad t=0..T

Fixed depth T; gradients flow through all T steps.

Implicit Fixed Point (DEQ, HRM, TRM)

z^* = f_theta(z^*, x)

Solve for z^* at test time; train with implicit differentiation (no backprop through time).

Key Methods

Method Fixed-Point Formulation Solver Depth
DEQ (Bai et al., 2019) z^* = f_theta(z^*, x) Anderson / Broyden Adaptive
HRM (2025) z_H^* = f_H(z_H^*, z_L^*), quad z_L^* = f_L(z_L^*, z_H^*, x) Iterative (H-steps) Hierarchical
TRM (2025) z^* = f_theta(z^*, x) (2-layer net) Simple iteration Flat, single module
Universal Transformer z^* = f_theta(z^*, x) Fixed T + ACT Adaptive (ACT)

TRM: Flat Recursion as Fixed Point

TRM uses a single 2-layer network f_theta applied recursively:

z_{k+1} = f_theta(z_k), quad z_0 = embed(x)

Limit z^* = f_theta(z^*, x) is the fixed point — the "answer" to the reasoning task.

Why it works: Deep supervision provides gradient at every step k:

L = sum_k lambda_k L_k(z_k, y)

This stabilizes convergence to the correct fixed point.

HRM: Coupled Fixed Points

HRM has two coupled fixed points:

begin{aligned} z_H^* &= f_H(z_H^*, z_L^*, x) quad &(High-level planning) \\ z_L^* &= f_L(z_L^*, z_H^*, x) quad &(Low-level execution) end{aligned}

ACT halting: H-module runs for T_H steps (Q-learned); L-module runs per H-step.

  • Adaptive depth: Hard problems → more H-steps
  • Deep supervision: Per-segment loss at each H-step

Training via Implicit Differentiation

For DEQ/flat recursion (TRM-style):

Forward: Solve z^* = f_theta(z^*, x) → z^* Backward: (∂ z^*)/(∂ theta) = ( I - (∂ f_theta)/(∂ z) big|_{z^*} )^{-1} (∂ f_theta)/(∂ theta)

Approximations: Conjugate gradient, Neumann series, Anderson acceleration.

Convergence & Deep Supervision

Without Deep Supervision With Deep Supervision
Converges to any fixed point Converges to task-relevant fixed point
Gradients vanish/explode Gradients at every step k
Hard to train >10 steps TRM trains 50+ steps easily

TRM finding: Deep supervision is the primary driver of performance; flat recursion + deep supervision > hierarchy + standard supervision.

Inference-Time Scaling via Fixed Points

Method Depth Control Compute
ACT Learned halting distribution Adaptive
HRM H-module steps (Q-learned) Adaptive
TRM Iteration count (fixed or ACT) Adaptive
DEQ Solver tolerance Adaptive

Unified view: All are fixed-point solvers with learned/adaptive termination.

Theoretical Guarantees

Property Condition
Existence f_theta is contraction (`
Uniqueness Banach fixed-point theorem
Convergence rate Linear (`
Implicit gradient (I - J_f) invertible at z^*

HRM/TRM: Deep supervision encourages f_theta to be contractive near solution.

Related Concepts

Sources