Reproduce “Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexey Kravets, Da Chen, Vinay P. Namboodiri

A critical analysis of existing few-shot CLIP benchmarks reveals that prevalent evaluation methodologies often overestimate inductive generalization by relying on CLIP's pre-existing knowledge from pre-training. Researchers developed a novel pipeline leveraging machine unlearning to create genuinely inductive benchmarks, demonstrating that most existing methods experience a substantial average performance reduction of 55% in this setting, while their proposed Self-Enhanced Prompt Tuning with Residual Textual Features (SEPRES) method shows superior robustness.

Key insights

  • Existing few-shot CLIP methods heavily rely on pre-existing knowledge from pre-training, leading to an average performance drop of approximately 55% when evaluated in a truly inductive setting.
  • Machine unlearning can effectively create inductive benchmarks by mimicking a model's state as if it had never encountered target classes, validated through 'oracle baselines'.
  • Introducing learnable residual textual features to prompt tuning (SEPRES) enhances a model's ability to acquire genuinely novel information from few-shot examples, making it more robust in inductive settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.