Reproduce “Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models” on OpenResearch

About this paper

Boyang Guo, Liang Li, Lin Peng, Yuhan Gao, Xichun Sheng, Chenggang Yan

A new prompt tuning framework, Cluster-Aware Neural Collapse Prompt Tuning (CPT), addresses the challenge of tail-class discriminability in vision-language models for imbalanced datasets. It achieves this by applying Neural Collapse principles locally within semantic clusters, improving average harmonic mean accuracy on new classes by up to 0.71% and notably reducing training instability across random initializations.

Key insights

• Applying Neural Collapse principles locally within pre-defined semantic clusters can effectively enhance tail-class discriminability without distorting the valuable global semantic hierarchy learned by pre-trained VLMs.
• Explicitly anchoring learnable textual prototypes to their frozen pre-trained counterparts (Rotation Stabilization Loss) significantly stabilizes training and reduces performance variance across different random initializations.
• A static, cluster-aware structuring of the embedding space provides a robust framework that simultaneously improves local class separability and preserves the VLM's inherent generalization capabilities across diverse tasks and domain shifts.

Summary by alphaXiv.