Reproduce “Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cristian Rodriguez-Opazo, Ehsan Abbasnejad, Damien Teney, Hamed Damirchi, Edison Marrese-Taylor, Anton van den Hengel
Researchers at the Australian Institute for Machine Learning (AIML) demonstrated that different CLIP vision backbones exhibit distinct strengths, developing the Neural Logit Controller (NLC), an adaptive ensembling method. NLC significantly improved zero-shot image classification accuracy by an average of 9.1% across 21 datasets, with gains up to 39.1% on EUROSAT, and enhanced few-shot performance while outperforming other ensemble techniques.
Key insights
- CLIP backbones, even when trained on identical data and objectives, develop distinct internal representations and demonstrate significant complementarity, with an 'ORACLE' showing potential gains over 40% on some datasets.
- Backbone architecture serves as the primary driver for prediction diversity, surpassing the diversity stemming from different pre-training datasets or training epochs.
- An input-conditional, adaptive logit weighting strategy is crucial for effectively combining the diverse strengths of multiple CLIP backbones, as evidenced by NLC's superior performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.