Reproduce “REVES: REvision and VErification--Augmented Training for Test-Time Scaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanxin Liu, Ruida Zhou, Xinyan Zhao, Amr Sharaf, Hongzhou Lin, Arijit Biswas, Mohammad Ghavamzadeh, Zhaoran Wang and 1 more
REVES is a post-training framework that enhances Large Language Models' (LLMs) ability for test-time sequential revision by learning from their 'near-miss' errors. The approach consistently improves performance on coding and mathematical reasoning tasks, generalizing to out-of-distribution puzzles, and matched state-of-the-art on circle packing with a smaller base model.
Key insights
- Improving an LLM's one-step revision value on sequential revision-induced inputs formally transfers to performance gains for any revision-using test-time scaling algorithm.
- A formal objective mismatch exists: optimizing the single-shot objective does not necessarily optimize the sequential revision objective, highlighting the need for specialized training.
- The sequential revision objective can be hazard-decomposed into a sum of per-state one-step recovery probabilities, enabling a sharp, state-specific training signal for local error identification and correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.