Reproduce “SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leo Fillioux, Omprakash Chakraborty, Ismail Ben Ayed, Paul-Henry Cournède, Stergios Christodoulidis, Maria Vakalopoulou, Jose Dolz
Semantic Orthogonal Calibration (SoC) introduces a Huber-loss based regularizer for Test-Time Prompt Tuning (TPT), addressing overconfidence in Vision-Language Models by enforcing class prototype separation more smoothly than prior methods. This approach reduces the average Expected Calibration Error (ECE) by 2.3 points and improves mean accuracy by 0.9 points compared to O-TPT across 11 fine-grained classification datasets, while maintaining robustness to distributional shifts.
Key insights
- A theoretical lower bound on softmax confidence is derived, directly linking worst-case cosine similarity among class prototypes to the minimum possible confidence, thus formalizing how reducing similarity influences confidence.
- First-order analysis reveals that O-TPT's full orthogonality constraint aggressively reduces worst-case class similarity, leading to a sharper increase in confidence than SoC's Huber-based regularizer, which applies gentler repulsion, especially for highly similar pairs.
- By capping repulsion for highly similar prototype pairs, SoC better preserves the inherent semantic structure among related classes in the embedding space, mitigating the over-separation observed in strictly orthogonal methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.