Reproduce “Closing the Curious Case of Neural Text Degeneration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, Ashish Sabharwal

Researchers from USC, Stanford, Saarland University, and AI2 provide a mathematical justification for truncation sampling methods, explaining how discarding low-probability tokens ensures sampled text aligns with the true language distribution. They also introduce Basis-Aware Threshold (BAT) sampling, which leverages the softmax bottleneck to refine token selection and improve text coherence.

Key insights

  • Truncation sampling works by guaranteeing that discarded tokens are not in the true distribution's support, based on a derived bound for model log-probability underestimation.
  • The softmax bottleneck's low-rank structure can be exploited to precisely determine if a token has a non-zero true probability, offering a "basis-aware" method for identifying plausible tokens.
  • It is possible to selectively discard high-probability, low-quality tokens while retaining high-quality, lower-probability tokens, a capability beyond traditional monotonic thresholding methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.