Reproduce “RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists” on OpenResearch

About this paper

Yuexi Yang, Alyssa Wu, Ji Luo, Richeng Xuan, Zhichao Hu, Yuhong Liu, Zhen Qin

The research introduces RepoProbe, a benchmark evaluating large language models' architecture-aware repository comprehension through open-ended questions derived from real GitHub Discussions. It proposes a checklist-based verification protocol to objectively assess LLM responses, demonstrating that frontier models often exhibit "Edit Bias" by prioritizing code modification over genuine architectural understanding.

Key insights

• "Edit Bias" is a prevalent failure mode across state-of-the-art LLMs, indicating models frequently default to proposing code modifications even when architectural understanding and design rationale are required.
• The Checklist-Based Verification Protocol offers superior stability and interpretability compared to traditional LLM-as-a-Judge scalar scoring, providing granular, actionable feedback for model improvement by identifying specific deficiencies.
• Fluent explanations from LLMs often mask a lack of deep technical correctness, as evidenced by consistently higher Clarity scores compared to Knowledge scores in evaluated models.
• Performance of LLMs varies significantly across programming languages and repository themes, with persistent weaknesses observed in less mainstream languages and certain challenging domains.

Summary by alphaXiv.