Reproduce “PRiSM: Prototype Regularization for Few-Shot VLMs” on OpenResearch

About this paper

Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz, Ismail Ben Ayed

ÉTS Montreal researchers developed PRiSM, a validation-free prototype regularization framework for few-shot Vision-Language Models, designed to correct biased and poorly separated class prototypes under realistic, imbalanced data conditions. This framework improved the mean accuracy of various baselines by up to +35.05% under severe class imbalance and consistently enhanced performance across different shot numbers and imbalance regimes.

Key insights

• Under realistic conditions of severe class imbalance and high effective classes, standard few-shot VLM methods experience significant performance degradation, which often worsens with increased supervision.
• The quality and bias of class prototypes are critical bottlenecks for few-shot VLM performance under imbalanced data, suggesting that robust prototype rectification is essential for effective adaptation.
• A theoretically grounded Lipschitz learning rate provides a robust and efficient step size for prototype updates, removing the need for empirical validation-based tuning in true few-shot scenarios.

Summary by alphaXiv.