Reproduce “Robust Calibration of Large Vision-Language Adapters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, Jose Dolz
This paper from ETS Montreal reveals that common CLIP adaptation strategies, while improving accuracy, significantly degrade model calibration, particularly on out-of-distribution data. The research identifies the expansion of logit ranges as the root cause and proposes simple, effective methods like Sample-Adaptive Logit Scaling (SaLS) to restore calibration while maintaining accuracy.
Key insights
- Popular CLIP adaptation strategies like Adapters, Prompt Learning (CoOp), and Test-Time Adaptation (TPT) significantly degrade model calibration, especially on out-of-distribution (OOD) data, despite improving accuracy.
- The primary cause of this miscalibration is the expansion of logit ranges (the difference between maximum and minimum pre-softmax scores) during adaptation, leading to overconfident predictions.
- Controlling the logit ranges to align with the zero-shot CLIP model's logit range is an effective strategy for restoring calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.