Reproduce “Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use” on OpenResearch

About this paper

Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, Christopher D. Manning

SWiRL trains models to make decisions across multi-step tasks by generating tool-use trajectories, splitting them into prefixes ending at successive actions, and scoring each action in context with a model-based reward. Its process filter keeps trajectories with actions judged reasonable, even when the final answer is incorrect. For Gemma 2-27B, HotPotQA training raised GSM8K model-judged accuracy from 0.65 to 0.76 on 300 examples.

Key insights

• Process-based filtering of synthetic data, which evaluates the reasonableness of each intermediate step, is more effective for multi-step reinforcement learning than filtering based on the final outcome.
• SWiRL demonstrates robust generalization capabilities, improving performance across different multi-step tasks (e.g., question answering to mathematical reasoning) and tool types (e.g., search to calculator) even when trained on a single domain.
• The method can effectively learn from trajectories with logically sound intermediate steps, even if the final answer is incorrect, showcasing a deeper learning of reasoning processes compared to Supervised Fine-Tuning (SFT).

Summary by alphaXiv.