Reproduce “Finetune like you pretrain: Improved finetuning of zero-shot vision models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, Aditi Raghunathan

Researchers from Carnegie Mellon University and Stanford University propose Finetune Like You Pretrain (FLYP), a method that reuses the original contrastive pretraining objective for supervised finetuning of zero-shot vision models like CLIP. This approach consistently achieves superior in-distribution accuracy and out-of-distribution robustness across various benchmarks compared to standard finetuning techniques.

Key insights

  • The choice of the contrastive loss function is crucial for FLYP's success, significantly outperforming cross-entropy alternatives in maintaining robustness.
  • Updating both the image and language encoders during finetuning is essential, as freezing the language encoder leads to degraded performance.
  • Precisely matching the pretraining process, including aspects like 'naive' handling of class collisions within batches, is key to achieving strong generalization and robustness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.