Reproduce “Transferring Knowledge from Large Foundation Models to Small Downstream Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shikai Qiu, Boran Han, Danielle C. Maddix, Shuai Zhang, Yuyang Wang, Andrew Gordon Wilson

Adaptive Feature Transfer (AFT) enables efficient and selective knowledge transfer from large foundation models to smaller downstream models by regularizing the downstream model to prioritize task-relevant pre-trained features. It consistently outperforms traditional methods like knowledge distillation and standard fine-tuning across various vision and language tasks, demonstrating robustness to noisy features and effective multi-source integration.

Key insights

  • AFT formulates knowledge transfer as a Bayesian inference problem, regularizing the downstream model to prioritize learning task-relevant subsets of pre-trained features.
  • The method selectively transfers information by learning weights for pre-trained features, enabling it to ignore irrelevant or noisy features.
  • AFT allows for combining knowledge from multiple pre-trained models, even with different architectures, by integrating their features in a principled manner.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.