Reproduce “MoPD: Mixture-of-Prompts Distillation for Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yang Chen, Shuai Fu, Yu Zhang

MoPD (Mixture-of-Prompts Distillation) presents a soft prompt learning method for Vision-Language Models that uses a dynamic selection of multiple hard prompts to enhance generalization to unseen classes. This approach achieves superior average harmonic mean accuracy across 11 benchmark datasets and maintains performance even with noisy prompt pools.

Key insights

  • Distilling knowledge from a diverse pool of multiple hard prompts offers richer guidance and is more effective for improving soft prompt generalization than relying on a single hard prompt.
  • Dynamic, instance-specific selection of teacher prompts via a trainable gating network significantly enhances the learning process by tailoring guidance to individual image characteristics.
  • The gating mechanism provides robustness to noisy prompts within the teacher pool, as it learns to prioritize informative prompts and effectively filters out irrelevant or low-quality ones.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.