Reproduce “Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting” on OpenResearch

About this paper

Sunny Sanyal, Hayden Prairie, Rudrajit Das, Ali Kavis, Sujay Sanghavi

Researchers at the University of Texas at Austin and Google Research developed FLOW, a method that upweights 'easy' samples during fine-tuning to mitigate catastrophic forgetting in a data-oblivious setting. This approach reduced the ImageNet-1K accuracy drop for a ResNet-50 from 42.11% to 2.93% and improved general capabilities for large language models, like Gemma 2 2B, by up to 10% on benchmarks such as MBPP.

Key insights

• Mitigating catastrophic forgetting can be achieved by focusing on the sample space, a different approach compared to most existing methods that operate in parameter or gradient spaces.
• Upweighting 'easy' samples, rather than 'hard' ones, can effectively preserve a pre-trained model's general capabilities by stabilizing its knowledge base during fine-tuning.
• Theoretical analysis suggests FLOW stalls learning along certain parameter directions, preventing overfitting to the new task and thus contributing to forgetting mitigation.

Summary by alphaXiv.