Reproduce “Reinforcement Learning for Long-Horizon Interactive LLM Agents” on OpenResearch

About this paper

Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, Philipp Krähenbühl

Apple Inc. researchers developed LOOP, an efficient reinforcement learning algorithm for long-horizon interactive large language model agents. This method achieved state-of-the-art performance on the AppWorld benchmark, surpassing models like OpenAI o1 by 9 percentage points in Task Goal Completion and fostering emergent complex behaviors such as improved API documentation consultation and error recovery.

Key insights

• Direct reinforcement learning in complex interactive environments can induce sophisticated emergent behaviors in LLM agents, including consistent API documentation consultation, reduced unwarranted assumptions, and improved error recovery.
• Defining PPO actions at the token level, rather than per-turn or per-trajectory, is critical for achieving stable training and superior performance in long-horizon interactive tasks.
• Memory-efficient RL, exemplified by LOOP's leave-one-out baseline and LoRA fine-tuning, makes training large LLM agents feasible without requiring a separate value network or multiple LLM copies, even with limited training data.

Summary by alphaXiv.