Reproduce “On the estimation and validity of AI time horizons---a statistical look at the METR plot” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Drew T. Nguyen, William Fithian
The authors revise METR’s software-task horizons by modeling how human task time maps to AI difficulty rather than assuming a linear relationship. They compare spline and item-response models with a shared-slope logistic baseline across 228 tasks and 26 AIs. In cross-validation, both proposed methods have better point estimates on all twelve scoring metrics, but individual-AI gains are mostly uncertain within this task suite.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.