Reproduce “MM-ACT: Learn from Multimodal Parallel Generation to Act” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotian Liang, Xinyi Chen, Bin Wang, Mingkang Chen, Yitian Liu, Yuhao Zhang, Zanxin Chen, Tianshuo Yang and 7 more
MM-ACT introduces a unified Vision-Language-Action (VLA) model that integrates text, image, and robot actions into a shared token space, enabling efficient parallel generation for real-time robotic control. This framework achieves state-of-the-art performance with an average 96.3% success rate on LIBERO simulation tasks and improved generalization to out-of-domain manipulation.
Key insights
- Jointly supervising task planning and future image prediction from a shared context significantly boosts action generation performance, with unified text-image-action training providing a +9.25% gain on RoboTwin2.0.
- A unified token space with parallel decoding across all modalities simplifies VLA model architecture and substantially improves efficiency for real-time robotic control.
- Different modalities exhibit distinct training dynamics (e.g., text overfitting faster than image), highlighting the necessity for careful loss weighting and scheduling in multimodal training pipelines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.