Reproduce “RepoQA: Evaluating Long Context Code Understanding” on OpenResearch

About this paper

Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang

RepoQA introduces the first benchmark designed to evaluate Large Language Models' capacity for long-context code understanding, featuring 500 tasks across 50 repositories and 5 programming languages. The benchmark demonstrates that top proprietary models achieve over 90% accuracy, while revealing that removing natural language comments from the code context often improves performance for most LLMs.

Key insights

• Top proprietary LLMs (e.g., Gemini-1.5-pro, GPT-4o) achieve over 90% accuracy in long-context code understanding, demonstrating a measurable lead over the best open-source models (e.g., DeepSeek-V2-Chat at 83.4%).
• Removing natural language comments from the code context surprisingly *improved* retrieval performance for most LLMs, suggesting comments might sometimes hinder, rather than aid, machine comprehension.
• LLMs' code understanding capabilities vary across programming languages, with generally better performance on Java and TypeScript compared to C++ and Rust, potentially correlating with training data distribution.

Summary by alphaXiv.