Reproduce “Prompt Coverage Adequacy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Florian Tambon, Michael Konstantinou, Cedric Richter, Charles Chenouard, Mark Harman, Mike Papadakis
The study proposes "Prompt Coverage Adequacy," a novel criterion for evaluating test suites against natural language prompts in LLM-driven software development, showing it detects over 30% more faults than code coverage-guided test generation on benchmarks.
Key insights
- LLM's internal mechanisms, such as attention and entropy, can be used to objectively measure the coverage of natural language requirements by test cases, analogous to code coverage.
- Prompt coverage provides a complementary, specification-based perspective on test adequacy that is distinct from and more effective than traditional code coverage for LLM-generated software.
- Shifting testing focus to natural language specifications offers robustness in iterative LLM development, as it remains relevant even when code structures change or implementations are unavailable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.