Reproduce “Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junyi Li, Xiaowei Qian, Yingyi Zhang, Wenlin Zhang, Guojing Li, Sheng Zhang, Xiao Han, Yichao Wang and 1 more

Researchers introduce ParetoPO, a two-stage multi-objective online reinforcement learning framework that optimizes tool-integrated language agents for both task accuracy and tool-use efficiency. The approach consistently achieves higher task accuracy while using significantly fewer tool calls across mathematical reasoning and multi-hop question answering benchmarks.

Key insights

  • Optimizing auxiliary objectives like tool-use efficiency can positively impact primary task accuracy by reducing redundant tool calls, mitigating noise, shortening trajectories, and enhancing credit assignment.
  • A two-stage approach, combining global exploration of the objective space with dynamic scalarization and local refinement via dominance-aware ranking, effectively navigates complex multi-objective landscapes.
  • Theoretical guarantees ensure that Stage 1 asymptotically covers supported Pareto-optimal points and Stage 2 converges to Pareto-ascent stationary points, providing a robust and stable optimization framework.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.