Reproduce “Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Yu, Yue Wang, Zihan Deng, Yan Zhu, Wenxiao Ma, Quanying Liu
Researchers at Southern University of Science and Technology demonstrate that large pretrained models develop a reusable spectral basis of leading singular vectors in their weight matrices, which remain highly stable and transferable during finetuning across diverse tasks and modalities. This spectral stability is linked to pretraining scale and enables a parameter-efficient finetuning method that achieves competitive performance by solely optimizing leading spectral coefficients.
Key insights
- Pretraining establishes a reusable spectral coordinate system: leading singular vectors of weight matrices are highly stable and transferable across diverse downstream tasks and modalities.
- The stability of these singular vectors is primarily influenced by the relative magnitude of finetuning updates and their directional alignment with existing pretrained components.
- Larger pretraining datasets lead to a more stable and transferable spectral basis, offering a geometric explanation for the benefits of scaling laws.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.