Reproduce “Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ismail Labiad, Matthieu Kowalski, Marc Schoenauer, Rémi Munos, Julia Kempe
The paper trains a small concept generator to propose problem-specific approaches for a frozen larger answer generator. Its trajectories receive reinforcement-learning rewards from solver correctness, while answer rollouts stay fixed. On a filtered hard-math set, a trained 7B guide raises pass@128 from 19.00% to 39.20% for a 32B solver; the result is specific to difficult mathematics and training is compute-heavy.
Key insights
- Concept guidance does not consistently outperform a fair repeated-sampling baseline when both use high-diversity decoding; the revised single-trajectory procedure is more effective because it creates several distinct exploration directions efficiently.
- The main benefit comes from problem-relevant semantic diversity: mismatching concepts across problems sharply reduces performance, while increasing the number of concepts improves pass@128 with little change in pass@1.
- Training can make a 7B concept generator more useful than an untuned 32B concept generator, suggesting that search-policy quality matters separately from the size of the model that generates the hints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.