Reproduce “LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning” on OpenResearch

About this paper

Zhen Hao Wong, Jingwen Deng, Runming He, Zirong Chen, Qijie You, Hejun Dong, Hao Liang, Chengyu Shen and 2 more

A reinforcement learning framework, LogicPuzzleRL, cultivates robust mathematical reasoning in Large Language Models by training them on a diverse suite of logic puzzles with verifiable rewards. This approach notably improved performance on out-of-distribution mathematical benchmarks, particularly on mid-difficulty problems, by fostering transferable reasoning strategies.

Key insights

• Reinforcement learning on a diverse suite of logic puzzles can fundamentally reshape an LLM's internal reasoning processes, cultivating generalizable subroutines like constraint propagation and multi-step deduction.
• A broad, multi-task training curriculum (All-Game RL) fosters more robust and transferable inference capabilities compared to training on individual puzzle types.
• The learned reasoning skills are most effective for mid-difficulty mathematical problems that require systematic elimination and chained inference, rather than very easy problems or those demanding highly specialized knowledge.

Summary by alphaXiv.