Reproduce “Explicit Inductive Bias for Transfer Learning with Convolutional Networks” on OpenResearch

About this paper

Xuhong Li, Yves Grandvalet, Franck Davoine

Researchers from Sorbonne universités introduce L2-SP and L2-SP-Fisher regularization techniques for transfer learning with convolutional networks, which explicitly preserve features learned during pre-training by penalizing deviations from initial parameter values. These methods consistently improved classification accuracy, for instance, achieving 84.2% on MIT Indoors 67 compared to 79.6% with standard L2, and better retain the functional roles of network units, especially with limited target data.

Key insights

• Explicitly biasing fine-tuned parameters towards their pre-trained values through L2-SP regularization is more effective for knowledge transfer than standard L2 weight decay.
• While L2-SP-Fisher improves mitigation of catastrophic forgetting for the source task, its impact on target task performance is comparable to that of plain L2-SP.
• L1-based and Group-Lasso-based penalties for retaining pre-trained values performed less effectively than L2-based approaches, likely due to optimization challenges with discontinuous gradients.

Summary by alphaXiv.