Reproduce “Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong and 16 more
Researchers at Fudan University and Meituan developed a two-player framework for LLM reasoning that uses dedicated critique models to provide step-level feedback, leading to performance improvements of up to 29% on challenging mathematical reasoning tasks. This approach enhances both test-time inference and training-time self-improvement by separating reasoning generation from error evaluation.
Key insights
- Separating the reasoning generation (actor) from the critique (critic) process overcomes the limitations of self-critique, allowing for more objective and focused evaluation.
- Automated data generation techniques can effectively create large, high-quality datasets for training specialized critique models.
- Difficulty-aware computation allocation during training-time supervision can effectively shift performance gains towards harder problems, addressing the 'tail-narrowing' issue.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.