Reproduce “Top-$nσ$: Not All Logits Are You Need” on OpenResearch

About this paper

Chenxia Tang, Jianchun Liu, Hongli Xu, Liusheng Huang

Researchers at the University of Science and Technology of China developed Top-nσ, a logit-based sampling method that leverages the bimodal distribution of pre-softmax logits to identify and retain informative tokens while filtering noise. This approach consistently outperforms traditional decoding and sampling methods in reasoning accuracy across various benchmarks, especially maintaining performance at higher temperatures.

Key insights

• Pre-softmax logit distributions consistently exhibit a bimodal pattern: a large Gaussian-distributed noisy region and a small set of highly informative outlier tokens.
• This statistical regularity allows Top-nσ to separate meaningful tokens from noise, maintaining a stable set of candidate tokens (nucleus) regardless of the sampling temperature.
• Higher sigma distances between the maximum logit and the mean of the noisy region indicate stronger model confidence, correlating with smaller optimal nucleus sizes.

Summary by alphaXiv.