Reproduce “MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee
MVP-LAM, a latent action model from Seoul National University and Microsoft Research Asia, learns action-centric latent actions by leveraging multi-view videos with a novel cross-viewpoint reconstruction objective. This approach yields latent actions with higher mutual information to ground-truth actions and improves robot manipulation success rates by up to 20.8% on the SIMPLER benchmark.
Key insights
- Maximizing the mutual information between latent actions and ground-truth actions requires reducing capacity spent on viewpoint-specific factors beyond actual state changes.
- Cross-viewpoint reconstruction explicitly discourages latent actions from encoding viewpoint-specific noise, forcing them to capture only the shared, intrinsic dynamics of the action.
- Including diverse multi-view human manipulation videos during training further enhances action-centricity by prioritizing motion as the dominant source of visual transition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.