Reproduce “CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

Researchers from Harbin Institute of Technology and Microsoft Research empirically demonstrate that CLIP models function as strong vision-language few-shot learners, achieving competitive zero-shot Visual Question Answering (VQA) with a two-step prompting method (TAP-C) and improving few-shot VQA performance by up to 10 percentage points using parameter-efficient BiNor fine-tuning.

Key insights

  • CLIP models possess inherent, yet previously inaccessible, few-shot learning abilities for VLU tasks that can be effectively unlocked through sophisticated prompt engineering.
  • The learned vision-language alignment in CLIP is robust enough to enable successful zero-shot cross-modality transfer, allowing knowledge from one modality to benefit tasks in another.
  • Parameter-efficient fine-tuning strategies like BiNor are crucial for adapting large multimodal models with limited data, preventing overfitting and achieving strong performance gains.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.