Summary: Deep Equilibrium Models (DEQ; Bai et al., 2019) reframe deep networks as fixed-point problems
z^* = f_theta(z^*, x)instead of fixed-depth unrolling. This enables infinite effective depth, constant memory training (via implicit differentiation), and adaptive compute at test time.
Standard Network vs DEQ
| Aspect | Standard Deep Net | DEQ |
|---|---|---|
| Depth | Fixed L layers |
Infinite (solve to tolerance) |
| Memory (train) | O(L) activations |
O(1) (implicit diff) |
| Compute (test) | Fixed L × block |
Adaptive (solver iterations) |
| Depth control | Architecture | Solver tolerance |
DEQ Formulation
Forward: Find z^* such that z^* = f_theta(z^*, x)
Solvers: Anderson acceleration, Broyden's method, Newton (with line search)
Typical structure:
f_theta(z, x) = Sigma(W z + V x + b)
or for vision:
f_theta(z, x) = ConvBlock(z) + InputInject(x)
Implicit Differentiation (Training)
Goal: Compute (∂ L)/(∂ theta) without backprop through solver steps.
Fixed-point condition: z^* = f_theta(z^*, x)
Total derivative:
(dz^*)/(dtheta) = (∂ f)/(∂ z) (dz^*)/(dtheta) + (∂ f)/(∂ theta)
Solve for gradient:
(dz^*)/(dtheta) = ( I - (∂ f)/(∂ z) bigg|_{z^*} )^{-1} (∂ f)/(∂ theta)
In practice: Use conjugate gradient / Neumann series to solve linear system — no unrolling.
DEQ Variants
| Variant | Structure | Application |
|---|---|---|
| DEQ (Bai et al., 2019) | Single f_theta |
Classification, NLP |
| Multiscale DEQ | Hierarchical f_theta^1, f_theta^2 |
Vision, long-range |
| DEQ-Transformer | Attention in f_theta |
Language modeling |
| HRM (2025) | Coupled f_H, f_L |
Hierarchical reasoning |
| TRM (2025) | Single f_theta (2-layer) |
Flat recursive reasoning |
Memory Comparison
| Model | Depth | Train Memory | Test Memory |
|---|---|---|---|
| ResNet-101 | 101 layers | High | High |
| Transformer-12L | 12 layers | High | High |
| DEQ | ∞ (solved) | Constant | Adaptive |
| HRM | 2 modules × ∞ (coupled) | Medium (deep sup) | Adaptive |
| TRM | ∞ (flat) | Low (deep sup) | Adaptive |
DEQ advantage: Train 1000-layer equivalent on 1 GPU.
Convergence Guarantees
| Condition | Guarantee |
|---|---|
f_theta is contraction (` |
|
Lipschitz f_theta with L < 1 |
Banach fixed point theorem |
| Deep supervision (HRM/TRM) | Stabilizes to task-relevant fixed point |
Without deep supervision: May converge to wrong fixed point (trivial/degenerate).
In HRM / TRM
| Model | Fixed Points | Deep Supervision |
|---|---|---|
| HRM | Coupled: z_H^* = f_H(z_H^*, z_L^*), quad z_L^* = f_L(z_L^*, z_H^*) |
Per H-step |
| TRM | Single: z^* = f_theta(z^*) |
Per iteration (primary driver) |
| DEQ | Single: z^* = f_theta(z^*, x) |
None (original) |
TRM finding: Deep supervision > hierarchy for convergence to correct fixed point.
Related Concepts
- fixed-point-recursion — Theoretical foundation
- hrm-reasoning — Coupled fixed points
- trm-recursive-reasoning — Flat fixed point
- deep-supervision — Stabilization mechanism
- implicit-differentiation — Training method
- memory-efficient-training — Benefit
Sources
- Bai et al., "Deep Equilibrium Models" (NeurIPS 2019)
- Bai et al., "Multiscale Deep Equilibrium Models" (NeurIPS 2020)
- fixed-point-recursion — Local wiki
- hrm-reasoning — HRM paper breakdown
- trm-recursive-reasoning — TRM paper breakdown