Reproduce “ExecRepoBench: Multi-level Executable Code Completion Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao and 4 more
This paper introduces EXECREPOBENCH, a new multi-level executable code completion benchmark designed to simulate real-world programming scenarios by evaluating models in repository-level contexts. The authors also present REPO-INSTRUCT, a specialized instruction corpus, and fine-tune Qwen2.5-Coder-Instruct-C, which demonstrates improved functional correctness and cross-language generalization in code completion.
Key insights
- Execution-based metrics like Pass@k are crucial for accurately evaluating code completion models, as string-based metrics do not consistently correlate with functional correctness.
- Providing repository-level context significantly improves the functional accuracy of generated code by allowing models to understand cross-file dependencies.
- Hybrid instruction tuning and multi-level grammar-based completion tasks enhance models' ability to generate functionally correct code at various granularities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.