Reproduce “Your Benchmark Is an Empirical Measure Over Difficulty” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang
The authors audit benchmark difficulty distributions for score mismatch and gaps that can obscure localized model improvements. They estimate normalized item difficulty, compare each benchmark with a same-size uniform reference, and measure empty intervals. Across 25 benchmarks, 10 have mismatch of at least 0.10 and 8 have gaps of at least 0.10. In three tests, BenchFill detects gains missed by controls.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.