Reproduce “Efficient Prediction of Pass@k Scaling in Large Language Models” on OpenResearch

About this paper

Joshua Kazdan, Rylan Schaeffer, Youssef Allouah, Colin Sullivan, Kyssen Yu, Noam Levi, Sanmi Koyejo

Researchers from Stanford University, the University of Toronto, and EPFL developed a statistically robust framework using a beta-binomial distribution and a dynamic sampling strategy to efficiently predict Large Language Model `pass@k` scaling. This approach achieves significantly improved predictive accuracy and up to 10 times lower Mean Squared Error compared to prior methods, especially when extrapolating performance for high `k` from limited sampling budgets.

Key insights

• Existing `pass@k` prediction methods, including log-log linear regression and discretized-beta fitting, are fundamentally flawed due to incorrect statistical assumptions and biases, leading to inaccurate extrapolations.
• Modeling problem difficulty with a beta-binomial distribution provides a more statistically sound and accurate way to estimate `pass@k` curves, especially in data-limited scenarios.
• Dynamically allocating sampling budget to harder problems is significantly more efficient for predicting `pass@k` at high `k` than uniform sampling, as rare successes from difficult problems primarily shape the high-`k` regime.

Summary by alphaXiv.