Reproduce “From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench” on OpenResearch

About this paper

Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng

MCR-Bench is introduced as the first benchmark for evaluating multi-round code review, simulating real-world iterative processes and dynamic defect states. Experiments reveal mainstream large language models struggle with defect state tracking and suffer performance degradation in deeper review interactions.

Key insights

• LLMs face substantial challenges in maintaining cross-round memory and integrating historical context, leading to noticeable performance degradation as the number of review rounds increases.
• Specific failure mechanisms, such as state–temporal misalignment (incorrectly flagging resolved defects) and cross-round defect forgetting, are primary contributors to both false positives and false negatives.
• Comment quality and accurate defect identification are complementary but not directly correlated; models can produce fluent comments without precise defect detection, indicating a need for joint optimization.

Summary by alphaXiv.