Reproduce “Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tristan Cinquin, Geoff Pleiss, Agustinus Kristiadi

An empirical study explores whether process reward model (PRM)-guided tree search improves large language model (LLM) mathematical reasoning over Best-of-N selection. Researchers found these more complex search methods do not yield statistically significant performance gains despite substantial increases in computational cost, with PRM reliability declining significantly for intermediate reasoning steps.

Key insights

  • The efficacy of advanced tree search methods is limited by the poor quality of existing Process Reward Models (PRMs), which do not accurately approximate the true value function for complex reasoning paths.
  • PRM reliability significantly degrades with increasing reasoning depth, indicating a credit assignment issue where intermediate steps are less accurately evaluated than near-terminal states.
  • Current PRMs demonstrate limited generalization capabilities, with their correlation to correctness being notably lower on out-of-distribution problems compared to in-distribution tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.