Reproduce “DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories” on OpenResearch

About this paper

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu and 10 more

Peking University and Alibaba Group introduced DevEval, a manually-annotated code generation benchmark that aligns with real-world code repositories by providing comprehensive dependency information and contextual data. Evaluation on DevEval revealed that leading large language models exhibit a substantial performance drop compared to traditional benchmarks, indicating struggles with handling long and heterogeneous code contexts for both functional correctness and dependency recall.

Key insights

• Large Language Models (LLMs) show a substantial performance drop on DevEval compared to traditional benchmarks, indicating a "context gap" in their ability to handle real-world code generation.
• Providing even limited "local file" code contexts significantly improves LLM performance in both functional correctness and the recall of relevant dependencies.
• LLMs continue to struggle with understanding and leveraging information from lengthy and heterogeneous contexts, particularly for cross-file dependencies.

Summary by alphaXiv.