Reproduce “InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning” on OpenResearch

About this paper

Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao and 2 more

InftyThink+ trains models to reason in bounded rounds, passing written summaries between rounds, with supervised fine-tuning followed by reinforcement learning across complete trajectories. For a 1.5B model, task-reward reinforcement learning raises AIME24 accuracy from 29.48% to 50.94%, versus 38.75% for vanilla long-chain-of-thought reinforcement learning; rollouts are capped at five iterations, so unlimited reasoning is not demonstrated.

Key insights

• Reinforcement learning, when applied to a structured iterative reasoning paradigm, significantly amplifies performance gains compared to vanilla RL, demonstrating a +9.89 point average accuracy improvement.
• Optimizing iterative decision-making (when/how to compress, how to resume) through RL leads to adaptive and effective reasoning strategies that outperform fixed heuristics or purely SFT-based approaches.
• A well-designed efficiency reward effectively balances reasoning accuracy with reduced inference latency and token generation, achieving an improved effectiveness-efficiency trade-off without significant performance collapse.

Summary by alphaXiv.