Reproduce “ViPRA: Video Prediction for Robot Actions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sandeep Routray, Hengkai Pan, Unnat Jain, Shikhar Bahl, Deepak Pathak
ViPRA learns latent motion tokens from unlabeled human and robot videos, predicting future visual states and multi-step scene changes. A flow-matching decoder maps these tokens to continuous robot-specific action chunks using labeled teleoperated demonstrations. On four SIMPLER Bridge tasks, ViPRA-FM reaches 62.5% average task-completion success, versus 53.2% without future-state prediction. It still requires demonstrations and 3.5 Hz real-world replanning.
Key insights
- Jointly predicting both future visual states ("what changes") and motion-centric latent actions ("how the scene evolves") during pretraining is crucial for learning grounded and transferable control priors.
- Pretraining on a diverse corpus of human and robot videos, rather than solely robot data, enhances generalization capabilities across different robot embodiments.
- Utilizing flow matching for continuous action generation, coupled with action chunking, provides stable and high-frequency control, essential for dexterous real-world manipulation, outperforming discrete action policies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.