Reproduce “Fine-tuning can cripple your foundation model; preserving features may be the solution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jishnu Mukhoti, Yarin Gal, Philip H.S. Torr, Puneet K. Dokania
The University of Oxford developed LDIFS, a regularization technique that addresses 'concept forgetting' in foundation models during end-to-end fine-tuning. This approach successfully maintains the models' broad conceptual understanding from pre-training and improves their performance in continual learning without compromising accuracy on specific downstream tasks.
Key insights
- Concept forgetting is a widespread issue in end-to-end fine-tuning of foundation models, leading to a significant reduction in their ability to recognize concepts outside the specific fine-tuning task.
- Preserving proximity in feature space, rather than just parameter space, is crucial for effectively mitigating concept forgetting and retaining the input-output behavior encoded by the pre-trained model.
- Knowledge encoded in intermediate layers of foundation models is vital; distilling information from concatenated features across these layers is more effective than using only the final layer's output for knowledge preservation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.