Reproduce “Ranking Reasoning LLMs under Test-Time Scaling” on OpenResearch

About this paper

Mohsen Hariri, Michael Hinczewski, Jing Ma, Vipin Chaudhary

Researchers at Case Western Reserve University formalized a dense benchmark ranking framework for reasoning LLMs under test-time scaling, systematically comparing 72 statistical methods. Their analysis revealed that while most methods agree at high trial budgets, a Bayesian estimator incorporating a greedy prior achieved the highest low-budget stability, reducing the standard deviation of Kendall's τb by 16–52% at N=1.

Key insights

• At high trial budgets (N=80), most statistical ranking methods produce orderings that closely align with a simple average accuracy gold standard, suggesting robustness in this regime.
• Incorporating an empirical prior from greedy decoding into Bayesian estimators (BayesR0@N) significantly reduces variance in low-budget rankings (16–52% τb std dev reduction at N=1) but can introduce bias depending on its alignment with stochastic sampling.
• Categorical ranking schemes can achieve high self-consistency but may show lower agreement with a purely correctness-based gold standard, indicating a trade-off when integrating auxiliary signals into evaluation.

Summary by alphaXiv.