Reproduce “SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Imbesat Hassan Rizvi, Xiaodan Zhu, Iryna Gurevych
The Single-Pass Annotation with Reference-Guided Evaluation (SPARE) framework automates fine-grained, step-by-step process supervision for large language models, leveraging existing reference solutions. It achieves up to 4.8% relative accuracy improvement in process reward model training and up to 2.4% relative F1 improvement in fine-tuning, while reducing annotation runtime by 2.6x compared to MCTS-based methods.
Key insights
- Fine-grained, step-level supervision generated by SPARE consistently improves LLM reasoning performance in both reward modeling and direct fine-tuning.
- A single-pass approach to jointly align and evaluate solution steps against references is significantly more computationally efficient than multi-pass search methods.
- Leveraging existing reference rationales for step-level evaluation reduces dependency on expensive human annotations or auxiliary models, enhancing data efficiency and generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.