Reproduce “Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adam Younsi, Ahmed Attia, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakim Hacid, Salem Lahlou

A framework combining an automatically trained Process Reward Model (PRM) with step-level Generative Flow Networks (GFlowNets) enhances Large Language Model mathematical reasoning. The approach improved accuracy on mathematical benchmarks, achieving a 9.4 percentage point gain on SAT MATH for a 3B model, and increased solution diversity compared to PPO-based methods.

Key insights

  • Automating PRM training via MCTS and similarity-based data augmentation effectively provides fine-grained, continuous feedback on reasoning steps without requiring costly human labeling.
  • Operating GFlowNets at the granularity of complete reasoning steps, rather than individual tokens, allows for more semantically coherent control and naturally promotes the generation of diverse, high-quality reasoning paths.
  • The reward-proportional sampling objective of GFlowNets is more effective than reward maximization (e.g., in PPO) for fostering both accuracy and diversity, leading to more robust and transferable mathematical reasoning skills.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.