Reproduce “Constructing Efficient Fact-Storing MLPs for Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Owen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré
A collaborative effort from Stanford, University of Wisconsin–Madison, and University at Buffalo researchers introduces a constructive framework for building Multi-Layer Perceptrons that efficiently store factual knowledge as key-value mappings for Transformer architectures. The proposed method achieves asymptotically optimal parameter efficiency and facilitates a modular approach to fact editing, demonstrably outperforming current state-of-the-art techniques.
Key insights
- The new `ρ(V)` (decodability) metric accurately predicts MLP fact-storage capacity, showing an R² > 97% across both constructed and GD-trained MLPs, highlighting the critical role of embedding geometry.
- A fundamental trade-off exists between an MLP's maximum fact-storage capacity (e.g., with whitened embeddings) and its usability within a Transformer, with higher Lipschitz constants correlating with reduced usability.
- A simple encoder-decoder mechanism, coupled with the Johnson-Lindenstrauss lemma for decoding, is sufficient to achieve near-information-theoretic optimal scaling for fact storage in MLPs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.