Reproduce “Optimizing Temperature for Language Models with Multi-Sample Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weihua Du, Yiming Yang, Sean Welleck
A method called TURN is introduced to automatically identify the near-optimal sampling temperature for large language models when using multi-sample aggregation, eliminating the need for labeled validation data. This approach leverages an intrinsic "entropy turning point" (EntP) to predict effective temperatures, achieving performance within 0.32% to 0.59% of grid-search optimum on math problem-solving and code generation tasks.
Key insights
- The optimal sampling temperature for multi-sample LLM inference varies considerably based on model architecture, training, and task, with fine-tuned models performing best at higher temperatures.
- An "entropy turning point" (EntP) exists in the log-scale token-level entropy curve, strongly correlating with the empirically determined optimal temperature for achieving peak accuracy.
- A simplified stochastic process model can reproduce the observed entropy spike, providing a theoretical explanation that EntP marks a rapid increase in "improper" tokens, signifying a collapse in sample quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.