Reproduce “SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks” on OpenResearch

About this paper

Timoleon Moraitis, Dmitry Toichkin, Adrien Journé, Yansong Chua, Qinghai Guo

This research introduces SoftHebb, an unsupervised learning algorithm for soft Winner-Take-All (WTA) networks that theoretically constructs a Bayesian generative model and minimizes cross-entropy using local Hebbian plasticity rules. The approach achieves competitive classification accuracy and demonstrates superior robustness to noise and adversarial attacks, alongside faster learning convergence compared to backpropagation-trained networks.

Key insights

• Unsupervised Hebbian learning in soft WTA networks can rigorously implement a Bayesian generative model, with local plasticity rules driving convergence to optimal parameters (centroids for weights, log-priors for biases).
• Local Hebbian plasticity enables implicit minimization of cross-entropy, bridging unsupervised learning with a common supervised loss function without global feedback or labels.
• Biologically plausible local learning can inherently confer superior robustness to noise and adversarial attacks, exhibiting "deflection" where attacks must perceptually alter input to deceive the network.

Summary by alphaXiv.