Reproduce “Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models” on OpenResearch

About this paper

Junjiao Tian, Chengyue Huang, Zsolt Kira

Researchers at Georgia Institute of Technology developed Selective Projection Decay (SPD), a weight decay technique that selectively regularizes model parameters during the fine-tuning of foundation models. This method consistently improved in-distribution generalization and out-of-distribution robustness across diverse tasks like image classification, semantic segmentation, and large language model reasoning, often outperforming more complex approaches.

Key insights

• Unconstrained fine-tuning with modern optimizers leads to excessive parameter deviation from the pre-trained state, harming robustness.
• Selective regularization, rather than uniform application, is critical for effectively balancing the preservation of pre-trained knowledge with adaptation to new data.
• The alignment between the negative gradient and the deviation from initialization provides a principled indicator for when to apply stronger regularization to specific layers.

Summary by alphaXiv.