Reproduce “Multimodal Policy Internalization for Conversational Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhenhailong Wang, Jiateng Liu, Amin Fazel, Ritesh Sarkhel, Xing Fan, Xiang Li, Chenlei Guo, Heng Ji and 1 more

A framework called TriMPI enables large multimodal models to internalize complex, reasoning-intensive multimodal policies, yielding up to 70.7% absolute performance gains in policy adherence while reducing inference prompt tokens by 93.9% and prefill time by 85.7%. It also improves generalization to policy updates and preserves general reasoning abilities.

Key insights

  • Internalizing multimodal policies through a multi-stage framework like TriMPI significantly improves policy adherence and reduces inference costs without compromising general reasoning or safety.
  • Reinforcement learning, particularly with the PolicyRollout (PoRo) mechanism for policy-aware exploration, is crucial for effectively internalizing reasoning-intensive policies, especially in low-data regimes.
  • The Visually-Masked Continual Pretraining (VM-CPT) stage enhances the model's ability to learn and ground complex multimodal instructions directly into its parameters.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.