Reproduce “Contrastive Action-Image Pre-training for Visuomotor Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuvan Sharma, Dantong Niu, Anirudh Pai, Zekai Wang, Zhuoyang Liu, Baifeng Shi, Stefano Saravalle, Boning Shao and 11 more
Contrastive Action-Image Pre-training (CAIP) trains a vision encoder by aligning egocentric human videos with human hand poses, which act as a proxy for robotic actions, through a contrastive learning objective. This approach yielded an average success rate of 76.04% on real-world dexterous manipulation tasks, surpassing other state-of-the-art vision encoders by over 30 percentage points, and demonstrated improved robustness to environmental variations.
Key insights
- Human hand poses in egocentric videos can effectively serve as a scalable and accessible proxy for robotic end-effector actions, enabling action-centric supervision from observational data.
- A contrastive learning objective is more effective than direct regression for learning robust, manipulation-relevant visual features from action-image pairs, leading to better downstream performance.
- Pre-training with explicit action signals induces visual representations that focus on task-relevant regions (e.g., hands, manipulated objects), which is crucial for effective visuomotor control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.