Reproduce “Combined Scaling for Zero-shot Transfer Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen and 4 more

Researchers at Google developed BASIC (Batch, Data and Model SIze Combined Scaling), a model for zero-shot transfer learning that achieves 85.7% top-1 ImageNet accuracy by combining massive scaling of data (6.6 billion image-text pairs), model size (3 billion parameters), and batch size (65,536). This approach yields a 9.3 percentage point improvement in zero-shot ImageNet performance and demonstrates enhanced robustness to distribution shifts.

Key insights

  • Comprehensive scaling across data, model, and batch size is a highly effective strategy for substantially improving zero-shot transfer accuracy and enhancing model robustness.
  • Very large contrastive batch sizes are essential for better generalization in image-text models, a finding supported by a theoretical framework indicating a reduced generalization gap.
  • Fine-tuning zero-shot models on target-domain labeled data can, counterintuitively, decrease their robustness to natural distribution shifts, challenging conventional views on generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.