Reproduce “Ubiquity of Emergent Hebbian Dynamics in Regularized Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Koplow, Tomaso Poggio, Liu Ziyin
Researchers demonstrate that Hebbian and anti-Hebbian synaptic plasticity signatures, widely observed in the brain, can emerge as generic properties of regularized and noisy learning dynamics in artificial neural networks. The work shows L2 weight decay promotes Hebbian alignment, while stochastic noise induces anti-Hebbian alignment, with a predictable phase transition governed by their interplay.
Key insights
- L2 weight decay acts as a contractive force that consistently drives the learning signal of various algorithms to align with Hebbian dynamics, even in non-learning systems.
- Stochastic noise in the learning process introduces an expansive force that can induce anti-Hebbian alignment, creating a dynamic competition with weight decay.
- A predictable phase transition boundary, characterized by a quadratic relationship ($\/gamma \/propto \/sigma^2$), determines whether Hebbian or anti-Hebbian dynamics emerge based on the balance between weight decay and noise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.