Reproduce “Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryo Karakida, Toshihiro Ota, Masato Taki
This research establishes a theoretical framework unifying entire MetaFormer blocks, including MLP-Mixers, with a three-layer hierarchical Hopfield network, termed "Energy MetaFormer". It demonstrates that while symmetric weights are inherent to the Hopfield formulation, spontaneous symmetry breaking is crucial for achieving high image recognition performance in Mixer models by resolving attractor degeneracy.
Key insights
- An entire MLP-Mixer/MetaFormer block can be fundamentally represented as a single hierarchical Hopfield network, offering a unified energy-based theoretical foundation for these architectures.
- Symmetric weight matrices, despite their origin in associative memory, lead to degenerate energy minima in the Hopfield network and substantially hinder performance in image recognition tasks for Mixer models.
- Spontaneous symmetry breaking in the weight matrices during training is crucial for MLP-Mixer models to achieve high performance, as it effectively resolves attractor degeneracy and enables the system to store and retrieve distinct patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.