Masked Visual Actions (MVA)
Summary: MVA is a pixel-space control interface that represents actions as partially revealed spatiotemporal patterns (binary masks over video). A single pre-trained video model finetuned with LoRA can perform both forward dynamics (simulate scene changes from robot motion) and inverse dynamics (recover robot motion from desired object motion) by varying which entities are masked vs. revealed. The model generalizes zero-shot to unseen robot embodiments and objects.
Core Idea: Actions as Visual Masks
Traditional robot control uses low-dimensional vectors (joint angles, end-effector poses). These are embodiment-specific — a policy trained on Franka joints doesn't transfer to a bimanual robot.
MVA instead represents actions in the same pixel space as the video model's training data:
Mask M(S) = bigcup_{i ∈ S} e_i
where e_i are entity pixel regions (robot arm, object). The world model predicts:
p(V mid M odot V, I_0)
Key insight: The direction of reasoning is determined by which entities are revealed:
- Forward: Reveal robot (
S=A) → predict objects (P) - Inverse: Reveal object goal (
S=P) → predict robot (A)
Architecture
| Component | Specification |
|---|---|
| Base Model | Wan-Fun-Control 2.2 (14B parameters) |
| Finetuning | LoRA (Low-Rank Adaptation) |
| Training Data | DROID (real), RoboCasa (sim) |
| Action Representation | Binary mask over video frames |
| Entity Segmentation | SAM (Segment Anything) + mesh rendering |
Two Data Generation Methods
- Segmentation-based (SAM): Auto-segment robot from DROID videos. Real-world diversity but potential mask leakage.
- Rendering-based: Robot meshes + camera calibration → clean "ghost" trajectories. Used for RoboCasa sim + calibrated real data.
Forward vs. Inverse Dynamics
Forward Model (Planning / Simulation)
p(\{e_i\}_{i ∈ P} mid \{e_j\}_{j ∈ A}, I_0)
- Input: Reference frame + robot trajectory (rendered or segmented)
- Output: Resulting object/scene motion
- Use cases: Model-based planning (sample N trajectories, pick best), policy evaluation (simulate policy without real execution)
Inverse Model (Control from Goals)
p(\{e_i\}_{i ∈ A} mid \{e_j\}_{j ∈ P}, I_0)
- Input: Reference frame + desired object trajectory
- Output: Robot motion achieving that trajectory
- Emergent capability: Trained only on forward (robot→object), but naturally inverts
- Human demo transfer: Extract object mask from human video → get robot motion
Key Results
| Benchmark | MVA | Ctrl-World (baseline) | Notes |
|---|---|---|---|
| DROID PSNR | 23.74 | 18.15 | Visual fidelity + control accuracy |
| Cross-embodiment (Franka → R1 Pro) | Works | Fails / static | BEHAVIOR-1K unseen robot |
| Novel gripper (custom 3D-printed) | Works | Fails | Real-world embodiment gap |
| Human demo → robot | Works | N/A | Inverse from human object motion |
Why MVA Generalizes Across Embodiments
| Representation | Embodiment Gap |
|---|---|
| Joint angles | High — different DOFs, kinematics |
| End-effector pose | Medium — same task space, different IK |
| Skeleton/keypoints | Medium — topology changes break it |
| MVA (pixel mask) | Low — visual appearance adapts naturally |
The video model learns visual physics (how pixels move), not robot kinematics. A new gripper just looks like a new shape moving — the model predicts its effect on surrounding pixels.
Related Work & Comparisons
| Method | Action Space | Forward | Inverse | Cross-Embodiment |
|---|---|---|---|---|
| MVA (this work) | Pixel mask | ✅ | ✅ (zero-shot) | ✅ Strong |
| Ctrl-World | Joint angles | ✅ | ❌ | ❌ |
| Diffusion Policy | Joint angles | ✅ | ❌ | ❌ |
| Genie | Latent actions | ✅ | ❌ | Partial |
| JEPA / LeWorldModel | Latent | ✅ | ❌ | Theory only |
Limitations & Open Questions
| Issue | Status |
|---|---|
| 14B params + 14B base model — heavy for real-time control | LoRA helps but still large |
| LoRA only — full finetuning not explored | May improve quality |
| Short horizons — videos ~16-32 frames | Long-horizon planning needs chaining |
| No explicit dynamics model — implicit in video | Hard to verify physical consistency |
| Segmentation dependency — SAM can fail | Rendering-based cleaner but needs calibration |
Connections to Wiki Concepts
- world-model — MVA is a video world model with action interface
- video-generation — Uses video diffusion backbone (Wan)
- diffusion-policy — Alternative action diffusion approach (joint space)
- joint-embedding-predictive-architecture — JEPA predicts in latent space; MVA predicts in pixel space
- genie — Latent action world model; MVA uses explicit visual actions
- embodied-ai — Core embodied AI contribution
- planning-in-latent-space — MVA plans in pixel space (video rollouts)
Research Idea Connections
- research_idea/mva-for-visual-reasoning — Apply MVA to visual reasoning tasks (seed)
- research_idea/mva-hierarchical-planning — Hierarchical MVA for long-horizon tasks (seed)
Sources
- source-masked-visual-actions-paper — arXiv:2607.19343 (Stanford, 2026)
- source-masked-visual-actions-blog — Project website with interactive demos
TABLE WITHOUT ID
file.link as Page,
type,
tags
FROM "wiki"
WHERE contains(file.path, "masked-visual-actions") OR contains(tags, "mva")
SORT file.mtime DESC