Reproduce “Dense Associative Memory for Pattern Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dmitry Krotov, John J Hopfield
Krotov and Hopfield introduce a dense associative memory model capable of storing patterns at a capacity that scales non-linearly with network size, `N^(n-1)`. This work establishes a duality between the memory's energy function and novel rectified polynomial activation functions, which improved MNIST classification accuracy and training speed while revealing a transition from feature-matching to prototype-based recognition.
Key insights
- Dense associative memories overcome classical capacity limitations, theoretically storing `K_max ≈ α_n N^(n-1)` patterns, significantly more than the `O(N)` of standard Hopfield networks.
- The simple duality provides a principled framework for deriving and understanding novel activation functions for deep learning directly from energy-based associative memory models.
- Varying the 'power of interaction vertex' (`n`) within the model smoothly transitions the network's recognition strategy from feature-based detection to prototype-based matching.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.