Reproduce “A Rubric-Supervised Critic from Sparse Real-World Outcomes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingyao Wang, Valerie Chen, Heng Ji, Graham Neubig
A semi-supervised learning framework trains a critic model for AI software engineering agents by combining sparse real-world outcomes with dense, trace-observable behavioral features called Critic Rubrics. This method enhances agent evaluation, inference-time efficiency, and training-time data curation, showing significant improvements in task success rates and robustness over traditional benchmark-only approaches.
Key insights
- Critics trained exclusively on academic benchmarks do not generalize to real-world agent interactions, underscoring the necessity of real-world data for practical critic development.
- Incorporating dense, process-based "Critic Rubrics" enables the critic to learn more robust and transferable failure mode representations, improving generalization across different LLM backbones.
- Code survival, a continuous, segment-level metric, provides more effective and granular supervision for critic training than coarser binary Pull Request merge status.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.