Reproduce “Cross-Embodiment Robot Foundation World Models with Latent Actions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat and 4 more
LAC-WM learns a shared motion representation from human and robot observations, then projects a new robot’s actions into it so a pretrained world model can predict outcomes. In simulated dexterous pick-and-place, it reaches 22% success versus 15% for the explicit-action baseline, a 46.7% relative gain. On modified LIBERO, it scores 92.0% versus 82.4%; these are simulated results, with no hardware demonstration.
Key insights
- Latent action embeddings cluster more closely across the three pretraining datasets than embeddings of explicit actions, and motion decoding improves this alignment.
- A shared latent action representation can make heterogeneous pretraining data more useful for a new embodiment: LAC-WM’s planning performance improves as the number of pretraining embodiments increases, unlike the explicit-action baseline in the reported experiments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.