Reproduce “On the Limits of Test-Time Compute for Self-Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anirudh Buvanesh, Khaled Rouissi, Sujin Yun, Johan Obando-Ceron, Morgane M Moss, Samuel Lavoie, Michael Noukhovitch, Laurent Charlin and 1 more

The authors introduce SCORPIO, which scores candidates in randomly formed groups, repeats judgments with new groupings, and averages ratings to select answers. The method reduces ranking variance faster in reported comparisons and improves cost-accuracy tradeoffs. On LiveCodeBench v6, GPT-OSS-20B with 128 candidates matches baseline peak top-1 accuracy at 18% lower inference cost, but does not close the gap to pass@N.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.