Reproduce “Outlier-Efficient Hopfield Layers for Large Transformer-Based Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo, Hong-Yu Chen, Weijian Li, Wei-Po Wang, Han Liu
Outlier-Efficient Hopfield Layers (OutEffHop) integrate a novel associative memory model into large transformer architectures to mitigate the outlier inefficiency problem. This approach achieves over 22% reduction in average activation kurtosis and enhances post-quantization performance across models like BERT, OPT, and ViT.
Key insights
- The Softmax₁ attention mechanism, noted for outlier efficiency, can be derived directly and formally from the energy landscape of a refined Hopfield model.
- OutEffHop theoretically guarantees monotonic energy minimization, achieves lower upper bounds on retrieval error, and possesses a larger exponential memory capacity than standard modern Hopfield models.
- By specifically associating "no-op" patterns with a zero-energy point, OutEffHop prevents them from influencing memory retrieval, providing a principled approach to outlier mitigation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.