Reproduce “SlimSearcher: Training Efficiency-Aware Web Agents via Adaptive Reward Gating” on OpenResearch

About this paper

Zequn Xie, Junjie Wang, Dan Yang, Jie Feng, Yue Shen, Jian Wang, Jinjie Gu

SlimSearcher, a training framework from Zhejiang University and Ant Group, integrates efficiency optimization into web agent training through a Multi-Stage Gating mechanism, reducing tool-call rounds by 17%-58% and lowering token consumption while maintaining or improving task accuracy across multiple long-horizon benchmarks. The approach guides models toward a "Minimal Necessary Path" by employing Pareto-efficient filtration during Supervised Fine-Tuning and Adaptive Reward Gating during Reinforcement Learning.

Key insights

• Strictly penalizing incorrect answers with a "correctness gate" is crucial to prevent reward hacking, where agents might otherwise generate vacuous short answers for high efficiency scores.
• Adaptive Efficiency Anchoring (AEA) provides a dynamic and stable efficiency signal by benchmarking against the empirically most efficient paths within an exploration batch, outperforming fixed penalties in balancing task efficacy and computational cost.
• Integrating efficiency optimization across both SFT and RL phases is necessary to systematically guide models towards a "Minimal Necessary Path," addressing both ingrained inefficient patterns and emergent over-exploration during learning.

Summary by alphaXiv.