Reproduce “Pareto optimal proxy metrics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alessandro Zito, Dylan Greaves, Jacopo Soriano, Lee Richardson
The authors select short-term proxies from existing metrics to balance sensitivity and agreement with the long-term north star. In a recommendation system, a proxy was tested on over 500 later experiments. It was 8.5 times as sensitive as the short-term north star, significant in 72% versus 40% of cases with a significant long-term north star, with no opposite-direction significant result in that holdout set.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.