Reproduce “RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use” on OpenResearch

About this paper

Jiajun Chai, Guojun Yin, Zekun Xu, Chuhuai Yue, Yi Jia, Siyu Xia, Xiaohan Wang, Jiwen Jiang and 4 more

RLFactory presents a plug-and-play reinforcement learning post-training framework designed to enhance Large Language Models' capabilities in multi-turn tool-use scenarios. It improved training throughput by 6.8 times and enabled a smaller LLM (Qwen3-4B) to achieve a test score of 0.486 on the NQ dataset, outperforming a larger model (Qwen2.5-7B).

Key insights

• Decoupling the training of tool invocation capabilities from diversified reward computation allows for more robust and adaptable integration of heterogeneous tools.
• Reconstructing the MDP state space with 'Observation Tokens' from external tool results establishes a dynamic, closed-loop interaction, crucial for effective multi-turn learning.
• A flexible, multi-paradigm reward system, encompassing rule-based, model evaluation, and tool verification approaches, is essential for addressing the varied evaluation needs of different multi-turn tasks.

Summary by alphaXiv.