Reproduce “RefineBench: Evaluating Refinement Capability of Language Models via Checklists” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck and 1 more
RefineBench introduces a comprehensive benchmark and checklist-based evaluation framework to assess the multi-turn refinement capabilities of language models across diverse, open-ended tasks. The study reveals that while frontier LMs struggle with autonomous error identification, they significantly improve when provided with explicit feedback, achieving near-perfect scores in guided refinement settings.
Key insights
- Even frontier LMs demonstrate significant limitations in autonomous self-refinement, often achieving only marginal improvements without external guidance.
- LMs are highly proficient at incorporating explicit corrective feedback, leading to substantial performance gains (e.g., Claude-Opus-4.1 improved by +79.7% in guided refinement).
- A primary bottleneck for self-refinement is the LMs' difficulty in autonomously identifying which aspects of their response need correction, rather than an inability to apply corrections once identified.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.