Reproduce “AgentRM: Enhancing Agent Generalization with Reward Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin and 2 more

AgentRM, a framework by researchers from Tsinghua University and Renmin University of China, employs a generalizable reward model to enhance the generalization of LLM-based agents to unseen tasks. It demonstrates that fine-tuning a reward model, particularly an explicit one, significantly improves agent performance and robustness across various agent tasks, achieving an average 8.8-point improvement over baselines and outperforming leading task-specific agents.

Key insights

  • Fine-tuning a reward model is more robust for generalization to unseen tasks compared to directly fine-tuning the policy model, which tends to overfit.
  • Explicit reward modeling, which directly learns task-level value functions, consistently provides the most significant performance improvements among the evaluated approaches.
  • A "weak-to-strong generalization" effect allows a reward model trained with data from a less powerful policy to significantly enhance the performance of stronger, more computationally intensive policy models in a plug-and-play manner.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.