Reproduce “Benchmark Prediction with Sparse and Unpaired Cached Responses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hayden Helm, Avijit Ghosh, Ben Johnson, Carey Priebe
The authors introduce PKPS to predict full-task scores from sparse evaluations or absent model–task results. It compares responses to similar rather than identical queries, then uses compact model representations and historical scores. On a selected EEE submatrix, PKPS achieved 0.070 MAE with one query per task cell, versus 0.224 for direct sample scoring; naturally sparse data were not tested.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.