Reproduce “Features are fate: a theory of transfer learning in high-dimensional regression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Javan Tahir, Surya Ganguli, Grant M. Rotskoff

This research analyzes transfer learning in high-dimensional regression, establishing that the utility of pre-trained models for new tasks depends primarily on the alignment between learned features and the target task, rather than statistical distances between source and target datasets. It quantifies transferability for deep linear networks, predicting when linear transfer and fine-tuning improve or degrade performance relative to training from scratch, and provides qualitative extensions to nonlinear models.

Key insights

  • Transferability is fundamentally determined by the learned feature space of the pre-trained model and its alignment with the target task, rather than solely by statistical distances between source and target datasets.
  • Pretraining in the feature learning regime can lead to feature sparsification, where the learned features predominantly represent the source task, potentially hindering transferability to dissimilar target tasks.
  • In deep linear networks, fine-tuning provides no transfer benefit in the underparameterized regime (as pre-trained memory is erased), but in the overparameterized regime, its success depends on the feature overlap, with positive transfer occurring for $\cos \theta > 1/2$.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.