Reproduce “Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation” on OpenResearch

About this paper

Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai and 2 more

Researchers from Shanghai Jiao Tong University and Meituan introduced PRDBench, an agent-driven benchmark with 50 project-level Python tasks across 20 domains, and PRDJudge, a specialized fine-tuned model for robust evaluation. PRDJudge demonstrated a human alignment rate exceeding 91% on complex project-level evaluations, outperforming larger general-purpose LLMs, and provided insights into the capabilities and costs of various code agents.

Key insights

• An iterative human-agent refinement process is effective for constructing high-quality, complex, project-level benchmarks with significantly reduced human expertise and annotation costs.
• Fine-tuning a specialized LLM for evaluation tasks leads to substantially higher human alignment and scoring stability compared to larger general-purpose LLM judges, enhancing the reliability of automated assessments.
• LLM code agents exhibit varied strengths in zero-shot generation versus debugging; commercial agents often show superior debugging capabilities despite sometimes constrained initial generation, while minimal agents face challenges in maintaining performance during long-context debugging.

Summary by alphaXiv.