Reproduce “Learning to Act from Actionless Videos through Dense Correspondences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, Joshua B. Tenenbaum
The Actions from Video Dense Correspondences (AVDC) framework enables robots to learn and execute tasks solely from actionless video demonstrations by synthesizing future video plans using a conditional diffusion model and inferring 3D robot actions from the dense optical flow between synthesized frames and initial depth information. This approach achieves 43.1% average success across 11 Meta-World manipulation tasks and demonstrates successful zero-shot transfer from human videos to robot control, all while significantly reducing computational training costs.
Key insights
- Inferring 3D actions from 2D optical flow between synthesized video frames and initial depth allows for policy learning and execution without requiring explicit action labels.
- Factorized spatial-temporal convolutions in diffusion models enable training high-fidelity video generation models on 4 GPUs in a single day, dramatically improving computational accessibility.
- A replanning strategy that re-synthesizes video plans based on current observations is critical for achieving robust, closed-loop control and mitigating accumulated errors in long-horizon tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.