Reproduce “LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Yang, Jiale Zhange, Jiawei Li, Hang Guo, Wen Huang, Jinpeng Wang, Peidong Liu, Shu-Tao Xia
LAWM-3D learns 3D-aware latent action representations from unlabeled multi-view human videos to enhance robot world models' generalization for embodied intelligence. The method demonstrates superior performance on out-of-distribution benchmarks and achieves improved action fidelity and spatial consistency in video generation.
Key insights
- Explicit geometric alignment and depth-based pixel-agnostic learning are crucial for training 3D-aware latent actions from heterogeneous multi-view human videos.
- Naive multi-view integration does not inherently lead to 3D understanding in latent action models; specific architectural and loss designs are necessary.
- Pretraining on diverse human video data, especially in combination with robot data, provides a better foundation for learning generalizable temporal representations for world models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.