Reproduce “LLM2: Let Large Language Models Harness System 2 Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, Wai Lam
Researchers from The Chinese University of Hong Kong and Tsinghua University developed LLM2, a framework integrating large language models with a process-based verifier, to provide real-time, token-level feedback during reasoning. This system consistently improved mathematical reasoning accuracy, for example, boosting Llama3-1B's GSM8K accuracy by 7.5 percentage points, and utilized a novel synthetic process-supervision strategy to train the verifier efficiently.
Key insights
- LLMs can be effectively endowed with System 2 reasoning capabilities at the token level through a process-based verifier, providing granular feedback without needing explicit intermediate verbalized reasoning.
- Synthetic generation of process-supervision data, by assessing the downstream quality of partial continuations, can overcome the data bottleneck for training effective token-level verifiers.
- Steering LLM generation with a verifier's score, combined with an adaptive plausibility constraint, enables significant performance gains with only a moderate increase in inference latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.