Reproduce “Calibration of Pre-trained Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shrey Desai, Greg Durrett
This research empirically examines the calibration of pre-trained Transformer models, BERT and RoBERTa, across diverse natural language understanding tasks and challenging out-of-domain scenarios. It demonstrates that these models possess reasonable out-of-the-box calibration, with temperature scaling significantly improving in-domain calibration and label smoothing reducing out-of-domain Expected Calibration Error by up to 5.8 times on adversarial datasets.
Key insights
- Pre-trained Transformers exhibit surprisingly good out-of-the-box calibration, particularly in out-of-domain settings, significantly outperforming non-pre-trained models.
- Temperature scaling is highly effective for improving in-domain calibration for MLE-trained models, suggesting that their logits are fundamentally well-structured.
- Label smoothing proves more effective for mitigating overconfidence and improving calibration in challenging out-of-domain scenarios, despite potentially hurting in-domain performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.