Reproduce “Temperature Fragility and the Conditional Benefits of Truncation Sampling” on OpenResearch

About this paper

Francesco La Rosa

An evaluation of 13 open-weight instruction-tuned language models shows that truncation samplers are most useful when higher temperatures trigger model-specific generation failures. Across GSM8K and MMLU-Pro, fragile models experienced accuracy drops of up to roughly 38 percentage points from temperature 0.7 to 1.3, while methods such as min-p and top-nσ often recovered much of the loss by reducing degenerate or unparseable outputs.

Key insights

• Temperature sensitivity is strongly model-dependent: six models lost about 17–38 percentage points on MMLU-Pro from 0.7 to 1.3, whereas seven others lost at most about 10 points. The effect persisted across inference engines and precision settings, and differed between Llama-3.1-8B and its post-trained variant Hermes-3-8B.
• For fragile models, rising-temperature errors were driven mainly by degeneration, token-budget capping, and unparseable outputs rather than by a comparable increase in coherent but incorrect answers.
• Truncation does not provide a uniform accuracy boost. Its value increases when plain sampling has already begun to break down, so reported sampler gains should always be interpreted relative to a same-temperature baseline.

Summary by alphaXiv.