Reproduce “Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model” on OpenResearch

About this paper

Jiang-Xin Shi, Chi Zhang, Tong Wei, Yu-Feng Li

Candle is a framework designed for efficient and robust adaptation of pre-trained vision-language models to long-tailed data distributions. It operates in the feature space, improving generalization to new classes and maintaining strong performance across diverse datasets and imbalance ratios while being computationally efficient.

Key insights

• Operating directly in the feature space enables efficient VLM adaptation and black-box compatibility, reducing computational overhead.
• Cross-modal attention enhances feature representation by leveraging complementary information between visual and textual modalities.
• Learnable virtual prototypes improve generalization to new classes by providing optimized representations for unseen categories, even with limited or no training data.
• The Compensating Logit-Adjusted Loss effectively addresses class imbalance within base classes and between base and new classes by treating virtual prototypes as training samples.

Summary by alphaXiv.