Reproduce “MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Roberto Garcia, Jerry Liu, Ronny Junkins, Sabri Eyuboglu, Atri Rudra, Christopher Ré
The authors introduce a bilinear Hebbian MLP for key-to-value fact storage. Its margin gives answers room to withstand query noise. For isotropic embeddings, it stores F facts with Θ(F log F) parameters. In synthetic one-layer Transformer tests with bounded attention noise, it uses 15–63× fewer parameters than the NTK baseline at matched fact counts; pretrained LLMs are not tested.
Key insights
- MLPs in Transformers can be formally understood as efficient Hebbian kernel memories, providing a theoretical framework for their knowledge storage capabilities.
- A sufficient decoding margin, bounded away from zero, is critical for MLPs to tolerate noisy inputs from attention layers and function reliably within Transformer blocks.
- Information-theoretically optimal fact storage capacity is achievable in MLPs even with arbitrary (anisotropic) input/output embedding geometries, with performance influenced by specific geometric penalties.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.