Reproduce “Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wachiravit Modecrua, Krittanon Kaewtawee, Krittin Pachtrachai, Touchapon Kraisingkorn
The authors develop a reward-calibration method combining IRC with discounted, dampened outcome signals to train tool-using customer-service agents. IRC uses rollout outcomes to assign rewards to action categories, while the hybrid checks advantage directions. In this 50-task airline evaluation, Qwen3.5-4B rose from 63.8% to 66.7%, and Qwen3-30B-A3B rose from 58.0% to 69.5%.
Key insights
- Dense per-turn rewards, when uncalibrated, can significantly degrade multi-turn agent performance due to "advantage misalignment" in advantage computation.
- Iterative Reward Calibration (IRC) reveals that empirically calibrated rewards, sometimes setting intuitive positive rewards to zero or negative, are essential for robust learning.
- Sparse reward training can accidentally focus gradients on critical "gold-diverse" positions due to "dead turns," which can sometimes outperform naïvely dense reward schemes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.