Reproduce “Predicting Code Coverage without Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michele Tufano, Shubham Chandel, Anisha Agarwal, Neel Sundaresan, Colin Clement
Researchers at Microsoft introduced a new benchmark task for Large Language Models (LLMs) to predict code coverage without execution, curating the COVERAGEEVAL dataset for evaluation. Their study found that GPT-4 achieved the highest Perfect Sequence Match at 30.04%, but all LLMs struggled significantly with branch correctness, reaching only about 22% for GPT-4.
Key insights
- LLMs demonstrate a nascent ability to predict code coverage, with multi-shot prompting generally improving performance across models.
- A significant gap exists in LLMs' understanding of code execution, particularly concerning conditional logic and precise control flow reasoning, as evidenced by low branch correctness scores.
- The Code Coverage Prediction Task serves as a robust benchmark for evaluating LLM code understanding, going beyond static code analysis to probe dynamic execution behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.