Reproduce “Fast Parametric Learning with Activation Memorization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jack W Rae, Chris Dyer, Peter Dayan, Timothy P Lillicrap
The paper introduces Hebbian Softmax, which helps neural classifiers learn rare classes by blending each active class’s output weight with the example’s hidden representation before ordinary gradient descent consolidates it. In Omniglot, it required fewer training steps after 80 classes, a reported 1–2× data-efficiency gain, but remained far from one-shot learning. On WikiText-103, test perplexity was 34.3 versus 36.4 for standard softmax, mainly benefiting less-frequent words.
Key insights
- Combining a fast, Hebbian-like memorization rule with gradient descent in the output layer enables efficient and rapid binding of novel classes into the network's parameters.
- A dynamic annealing schedule is crucial for balancing fast learning of new concepts with stable consolidation of frequently observed classes, preventing catastrophic forgetting.
- Parametric activation memorization offers a computationally efficient and more stable alternative to non-parametric caches for retaining information about infrequent items over longer time intervals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.