Reproduce “CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, Robby T. Tan
This work introduces CodeJudgeBench, a benchmark to evaluate LLM-as-a-Judge capabilities for coding tasks, encompassing code generation, repair, and unit test generation. It demonstrates that 'thinking models' outperform non-thinking models, while revealing considerable positional bias and the importance of full response context for accurate code judgment.
Key insights
- LLMs employing advanced reasoning techniques ('thinking models') consistently and significantly outperform non-thinking models in code judgment tasks, even with smaller parameter sizes.
- LLM-as-a-Judge models exhibit substantial positional bias, with performance varying by up to 14% simply due to the order of candidate responses.
- Providing the full raw model response, including comments and reasoning, is more effective for LLM judges than using only extracted code segments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.