Reproduce “Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, Tieniu Tan
This research re-evaluates reasoning prompting strategies under test-time compute scaling in large language models, demonstrating that simpler methods like Chain-of-Thought (CoT) often outperform more complex ones as sampling times increase, attributed to their favorable question difficulty distribution and uniform error probabilities. The study also introduces an O(1) probabilistic method to predict scaling performance and proposes adaptive strategies for improved efficiency.
Key insights
- Simpler prompting strategies, particularly Chain-of-Thought (CoT), consistently outperform more complicated strategies as test-time compute (sampling time N or computational cost O) scales across various LLMs and benchmarks.
- CoT's superior scaling performance is due to a higher proportion of 'easy' questions and a more uniform probability distribution among incorrect answers, which benefits more from majority voting.
- A strategy with lower initial accuracy can eventually surpass one with higher initial accuracy when scaled, depending on the probability spread between the correct and highest incorrect answers.
- Dynamically adjusting scaling based on question difficulty and selecting the optimal prompting strategy per question offers substantial performance gains, suggesting the limitations of a single, fixed prompting approach.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.