Why text-to-video is a harder problem than text-to-image

Adding a time axis to generation doesn't just multiply the output size — it multiplies the ways a model can be locally correct and globally wrong. A model can generate a plausible frame at t=0 and a plausible frame at t=1 that don't actually depict continuous motion of the same scene. Naively extending an image diffusion or autoregressive architecture to video tends to spend enormous compute modeling fine spatial detail at every timestep, when the harder problem is actually coarse temporal consistency: does this stay the same object, moving coherently, across the whole clip.

The hierarchical split

TTV-HRM's core structural choice is separating "what happens" from "what it looks like" into different levels of a hierarchy:

  • A coarse, high-level reasoning stage plans scene structure and motion over the full clip — object placement, camera movement, how the scene evolves — at a resolution too coarse to be visually detailed but cheap enough to reason over the entire temporal span at once.
  • A fine-grained rendering stage fills in per-frame visual detail conditioned on that coarse plan, so it only has to be locally consistent with a plan that's already temporally coherent, rather than solving temporal coherence and visual fidelity simultaneously at every layer.

This mirrors how hierarchical reasoning shows up elsewhere: solve the coarse structure of a problem first, then solve the details inside that structure, rather than trying to do both at the same resolution simultaneously.

Why this improves efficiency

Full spatiotemporal attention over every frame at every layer scales badly — the compute grows with the product of spatial resolution and clip length. By pushing full-sequence reasoning down to a coarse representation and only doing expensive fine-grained modeling within a much smaller effective context, the architecture spends its heaviest compute where it's actually needed for visual fidelity, and its cheaper compute where the problem is really about long-range consistency rather than detail.

What this trades away

Hierarchical decomposition is only as good as the coarse plan. If the high-level stage commits to a scene structure that turns out to be wrong or under-specified, the fine-grained stage has no path to fix it — it's conditioned on the plan, not free to override it. That's the standard failure mode for hierarchical generation methods generally, and it's the main place this architecture's output quality is bounded by the coarse stage rather than the fine stage.

Relationship to the broader efficient-transformer line of work

This sits alongside recurrent-depth and other test-time-compute-scaling approaches as a case of the same underlying strategy: don't spend uniform compute across a sequence, spend compute where the problem is actually hard. Here that strategy is applied along the hierarchy axis (coarse-then-fine) rather than the depth axis (shallow-then-deep).