Reproduce “CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li and 2 more
Researchers from Peking University and Huawei Cloud Computing introduced CoderEval, a new benchmark designed to evaluate large language models on pragmatic code generation by including context-aware functions with varying dependencies. This benchmark demonstrates that models perform substantially worse on non-standalone functions, which constitute over 70% of real-world code, highlighting a critical gap in current evaluation methodologies.
Key insights
- A substantial performance gap exists for LLMs between generating standalone functions and context-dependent (non-standalone) functions, validating the need for more pragmatic benchmarks.
- The ability of a model to correctly incorporate `oracle_context` information (measured by `Acc@K`) directly correlates with its overall functional correctness (`Pass@K`), emphasizing the importance of context understanding.
- Challenges in generating contextual tokens are language-specific; for instance, Python models struggled with `VarReference` tokens, while Java models struggled with `TypeReference` tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.