Reproduce “Calibrated Language Models and How to Find Them with Label Smoothing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jerry Huang, Peng Lu, Qiuhao Zeng

Research from Université de Montréal and Mila investigates the degradation of confidence calibration in large language models after instruction tuning, showing that label smoothing consistently reduces calibration error while maintaining accuracy. A novel, memory-efficient Triton-based computational kernel makes label smoothing practical for large vocabulary models by reducing memory consumption by over 95%.

Key insights

  • Supervised fine-tuning degrades LLM calibration by reducing the diversity of learned features, leading to increased overconfidence.
  • Label smoothing (LS) improves calibration by promoting a more uniform distribution over output labels and enhancing the separability and diversity of individual samples, equivalent to MAP estimation.
  • LS's effectiveness is diminished in models with smaller hidden dimensions but large vocabularies, as these models are inherently less prone to extreme overconfidence due to a higher entropy lower bound in their LM head predictions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.