Reproduce “Contextual Temperature for Language Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, Da-Chang Juan

Researchers from National Tsing Hua University and Google introduced a learnable, context-dependent temperature scaling mechanism for neural language models, dynamically adjusting output probabilities for individual tokens. This method improved perplexity on Penn Treebank to 47.42 and WikiText-2 to 40.96 with dynamic evaluation, while enhancing the semantic coherence of generated text.

Key insights

  • The optimal temperature for predicting a word is highly context-dependent, with different tokens developing distinct 'heating-up' or 'cooling-down' schedules during training.
  • Model uncertainty, as reflected by the learned temperature, is generally higher at the beginning of a sentence (due to less context) and gradually decreases towards the end.
  • Contextual temperature provides a theoretical advantage by allowing more flexible and aggressive parameter updates through dynamic control over gradient magnitudes for specific ground-truth classes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.