Reproduce “LeWAM: End-to-End World and Action Modeling with JEPAs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
LeWAM jointly learns how actions change a scene and which actions are plausible, then uses proposed actions to plan manipulation. Its model predicts latent states and action distributions, refining candidate sequences against a goal. Across ten simulated tasks, it averages 93.5% success versus 50.9% for LeWM with CEM. These results concern task-specific goal reaching from offline data, not broad transfer.
Key insights
- Joint action modeling appears to make visual representations more useful for control; probes show improved action and proprioceptive information compared with a dynamics-only JEPA baseline.
- Stochastic flow-matched action proposals can represent different valid behaviors, while SIGReg helps preserve useful latent structure and planning performance.
- Policy-guided refinement can reduce the need for broad population search, with particularly large gains reported on contact-rich simulated tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.