Reproduce “Agent Explorative Policy Optimization for Multimodal Agentic Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Minki Kang, Shizhe Diao, Ryo Hachiuma, Sung Ju Hwang, Pavlo Molchanov, Yu-Chiang Frank Wang, Byung-Kwan Lee

Agent Explorative Policy Optimization (AXPO) is introduced to bridge the "Thinking-Acting Gap" in multimodal agentic reasoning, a challenge where standard reinforcement learning under-trains tool use. This method notably improves vision-language model performance in tool-intensive tasks, with an 8B AXPO-trained model achieving superior Pass@4 scores compared to a 32B baseline.

Key insights

  • The "Thinking-Acting Gap" is characterized by under-attempted and frequently incorrect tool use, where standard RL methods fail to provide adequate learning signals for tool-call tokens.
  • Targeted exploration focused on the tool-call boundary, achieved through prefix-fixed resampling, is more effective than increasing overall rollout budgets or uniform exploration for improving tool-using capabilities.
  • By concentrating stochasticity and applying separate advantage calculations for shared prefixes and resampled continuations, AXPO can effectively restore positive learning signals for tool-using behaviors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.