Reproduce “Scaling Laws for Associative Memories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vivien Cabannes, Elvis Dohmatob, Alberto Bietti
Researchers at FAIR, Meta, and the Flatiron Institute developed a simplified associative memory model to provide a mechanistic explanation for scaling laws in Large Language Models. The study quantifies two error sources (data scarcity and memory capacity), identifying an optimal thresholding scheme for memory storage, and derives a combined generalization error scaling of E ~ T^(-1+1/α) + d^(-α+1).
Key insights
- Generalization error in associative memories is additively composed of data scarcity error (scaling with T^(-1+1/α)) and memory capacity error (scaling with d^(-α+1) for optimal storage).
- For models with random embeddings, a "thresholding" scheme (prioritizing storage of the most frequent P associations, where P scales with d) is the most efficient way to utilize finite memory capacity d.
- Optimization techniques like small batch sizes, adaptive optimizers (Adam), and Layer Normalization improve memorization by effectively normalizing or clipping gradients, leading to more uniform and efficient utilization of memory capacity.
- Learning input and output embeddings, rather than using fixed random ones, can dramatically increase memory capacity from linear to exponential in d, highlighting their critical role in real neural networks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.