Reproduce “Latent Action Pretraining from Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan and 8 more
LAPA learns a discrete vocabulary of visual changes from frame pairs, then trains a vision-language model to predict those changes from images and task descriptions. A smaller dataset of action-labeled robot trajectories maps the predictions to executable actions. On three real-world tasks, Open-X-pretrained LAPA averages 50.09% partial success versus OpenVLA’s 43.87%; this is not a win on every task.
Key insights
- Learning discrete latent actions from raw video observations allows for unsupervised pretraining without explicit action labels or pre-defined action priors.
- A shared, discrete latent action space can be learned that is robust across different robot embodiments and can transfer from human manipulation videos to robot tasks.
- Leveraging a pre-trained Vision-Language Model as a backbone and learning a small, discrete latent action space significantly improves pretraining efficiency (30-40x faster than prior methods).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.