Reproduce “Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenyou Fan, Fangzheng Yan, Chenjia Bai, Jiepeng Wang, Chi Zhang, Zhen Wang, Xuelong Li
CogRobot is a framework for learning generalizable bimanual manipulation policies that leverages pre-trained Text-to-Video (T2V) models by introducing optical flow as an intermediate representation. The framework achieves improved video generation quality (e.g., 22.663 PSNR on RDT) and higher success rates in bimanual tasks across both simulation (e.g., 100% for "Put Apple Cabinet") and real-world scenarios (e.g., 0.75 for "Pull box").
Key insights
- While bimanual actions are complex, their resulting state trajectories can be uniformly represented through videos, a domain where large T2V models excel.
- Optical flow serves as an effective intermediate representation, bridging the modality gap between text instructions and precise robotic movements by capturing fine-grained motion.
- Adapting pre-trained large-scale generative models, like T2V models, for embodied tasks can reduce reliance on extensive domain-specific robot data and improve physical grounding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.