Reproduce “MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yangcheng Yu, Xin Jin, Yu Shang, Xin Zhang, Haisheng Su, Wei Wu, Yong Li
MOWM introduces a framework for embodied action planning that integrates pixel-space and latent-space world models through latent-to-pixel feature modulation, creating a hybrid visual representation. It achieves state-of-the-art task success rates on the CALVIN benchmark and demonstrates effectiveness in real-world complex manipulation tasks, such as cloth folding.
Key insights
- A hybrid visual representation, combining fine-grained pixel details with motion-aware latent features, significantly enhances performance in embodied action planning compared to relying on either representation alone.
- Latent-space world models exhibit an inductive bias towards action-centric information, effectively encoding dynamic motion and geometric structures while potentially abstracting away task-irrelevant visual attributes like exact chromatic details.
- Simple concatenation-based feature fusion with a residual integration mechanism effectively leverages the complementary strengths of pixel and latent representations, outperforming more complex methods like cross-attention for this specific task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.