Reproduce “Modular Meta-Learning with Shrinkage” on OpenResearch

About this paper

Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew W. Hoffman, Nando de Freitas

The authors introduce a meta-learning method that learns which modules of a model should adapt to a new task. It uses learned Gaussian prior variances to shrink task-specific parameters toward shared values. In scarce-data Omniglot experiments, shrinkage methods outperform corresponding baselines. On reduced WaveNet TTS, σ-Reptile scores 2.28 ± 0.06 MOS after 8 minutes, but the model does not produce production-level samples.

Key insights

• Module-specific variance parameters (shrinkage priors) provide a principled mechanism to automatically differentiate between task-specific (high variance) and task-independent (low variance) components of a neural network.
• This Bayesian shrinkage effectively regularizes the adaptation process, preventing overfitting when large models are fine-tuned with minimal task-specific data.
• Established meta-learning algorithms such as MAML, iMAML, and Reptile can be unified and generalized as special cases within this modular, shrinkage-based framework.

Summary by alphaXiv.