Reproduce “See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangyan Chen, Meiling Wang, Qi Shao, Zichen Zhou, Weixin Mao, Te Cui, Minzhao Zhu, Yinan Deng and 5 more
A Vision-Language-Action (VLA) model named ViVLA enables robots to acquire novel manipulation skills from single expert video demonstrations at test time. Developed by Beijing Institute of Technology and LimX Dynamics, it achieves over 30% higher success rates on unseen tasks by leveraging a unified latent action space and a scalable data generation pipeline from human videos.
Key insights
- Learning a unified and semantically consistent latent action space through cycle-consistency is crucial for effective cross-embodiment knowledge transfer from various video sources.
- Synthesizing high-quality, diverse expert-agent paired data from readily available human videos via 3D Gaussian Splatting can overcome data scarcity and significantly enhance model generalization.
- Employing parallel decoding for action prediction, along with temporal-spatial masking of video inputs, is effective in preventing shortcut learning and improving the efficiency and robustness of VLA models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.