Reproduce “Meta Learning Backpropagation And Improving It” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Louis Kirsch, Jürgen Schmidhuber

Variable Shared Meta Learning (VSML) is introduced as a principle for neural networks to meta-learn powerful, generalizable learning algorithms (LAs) entirely within their recurrent dynamics, circumventing fixed backpropagation in the inner loop. The method successfully cloned backpropagation and demonstrated meta-learned LAs that generalized across diverse, unseen datasets, achieving faster learning than online gradient descent in various settings.

Key insights

  • Fundamental learning algorithms, including complex credit assignment mechanisms like backpropagation, can be implicitly learned and optimized within an end-to-end differentiable neural system.
  • Extensive variable sharing (V_L >> V_M) in VSML effectively addresses the overfitting issues prevalent in standard Meta RNNs, leading to improved generalization of meta-learned LAs.
  • Meta-learned LAs can discover qualitatively different and more sample-efficient learning strategies, such as "fast association," compared to traditional gradient descent methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.