Reproduce “APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning” on OpenResearch

About this paper

Guiming Cao, Kaize Shi, Hong Fu, Huaiwen Zhang, Guandong Xu

This paper introduces APLe, a framework for multi-modal prompt learning that addresses the challenges of disparate modality sophistication and prompt-length induced overfitting in Vision-Language models. APLe achieves this through a two-stage sequential training and adaptation process for vision and language prompts, along with an image pre-processing adapter, leading to improved generalization and robustness across various datasets and domain shifts.

Key insights

• Treating vision and language prompts as distinct entities initially, followed by a synergy-promoting adaptation stage, addresses the 'disparate sophistication' between modalities, enhancing overall learning.
• Integrating CLIP's zero-shot knowledge via KL divergence during sequential training, with modality-specific weighting, helps to escalate generalization and mitigate overfitting.
• Pre-processing images with an image adapter using FFT and a Gaussian filter stabilizes visual features by suppressing noise, making them more amenable to prompt learning.

Summary by alphaXiv.