Reproduce “RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qiaoyu Tang, Hao Xiang, Le Yu, Bowen Yu, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun and 1 more
RefCritic trains Chain-of-Thought critic models using a novel dual-reward reinforcement learning framework, enabling them to provide actionable feedback that improves policy model refinement on complex reasoning tasks. The method achieves notable gains in solution accuracy across math, coding, and QA benchmarks, and demonstrates strong capabilities in fine-grained error localization without requiring step-level annotations.
Key insights
- Supervised Fine-Tuning (SFT) alone is insufficient for training critics that provide actionable feedback, often leading to superficial critiques despite lengthy outputs.
- Explicitly linking a critic's reward to the downstream improvement of a policy model's performance (via refinement feedback) enables the critic to generate truly actionable and in-depth analyses.
- A critic trained with solution-level refinement feedback can implicitly learn to localize fine-grained errors within multi-step reasoning, even without explicit step-level supervision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.