Reproduce “DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation” on OpenResearch

About this paper

Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang, Fengjun Zhang, Chun Zuo

This research introduces DeepCRCEval, a comprehensive framework using nine quality criteria to re-evaluate Code Review Comment Generators (CRCGs), moving beyond traditional text similarity metrics. It also proposes LLM-Reviewer, a training-free, Large Language Model-based baseline, which generates objectively superior and more actionable comments than existing state-of-the-art CRCGs, as validated by both human and LLM evaluators.

Key insights

• Existing benchmark comments used for training and evaluating CRCGs are of questionable quality, with a significant portion (only 3-8% deemed ideal) lacking clarity, actionability, and specific problem identification.
• The DeepCRCEval framework provides a more discriminating, comprehensive, and efficient evaluation of CRCGs compared to traditional text similarity metrics, with LLM evaluators showing high concordance (ICC > 0.75) with human assessments.
• LLM-Reviewer consistently outperforms state-of-the-art CRCGs across all nine quality criteria, highlighting that prior models, guided by indirect text similarity, have likely overestimated their actual contribution to code quality and defect detection.

Summary by alphaXiv.