Reproduce “Construct Validity of Coding Benchmarks for LLM-based Agentic Coding Tools” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Berndt, Paul Ralph, Thomas Bach, Sebastian Baltes
The authors examine how coding benchmarks define and report what their scores measure through qualitative analysis of benchmark publications. For RQ1, eight benchmarks were analyzed: only two state constructs explicitly, and seven report task pass rates. For RQ2, seven benchmarks report task attributes inconsistently; repositories are most common, appearing in six. The findings identify validity threats, not proof of invalidity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.