Reproduce “Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shivansh Patel, Shraddhaa Mohan, Hanlin Mai, Unnat Jain, Svetlana Lazebnik, Yunzhu Li
RIGVid turns a scene image and language command into a candidate video, then converts the manipulated object’s motion into a robot trajectory. It estimates depth, tracks the object’s six-dimensional pose, and retargets that path to the robot. On the spatula task, filtered Kling v1.6 videos achieved 90% success versus 50% unfiltered; results cover four fixed-setup tasks.
Key insights
- Current state-of-the-art video generation models, when effectively filtered, can provide sufficient information for real-world robotic manipulation tasks without any physical demonstrations or robot-specific training.
- Dense, structured 6D object pose trajectories extracted from generated videos are more robust for robot execution than sparser VLM-based abstractions or point/flow-based tracking methods.
- The accuracy of monocular depth estimation is a primary bottleneck for overall robot performance, even when using high-quality generated videos and robust pose tracking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.