Reproduce “Exploring Biologically Inspired Mechanisms of Adversarial Robustness” on OpenResearch

About this paper

Konstantin Holzhausen, Mia Merlid, Håkon Olav Torvik, Anders Malthe-Sørenssen, Mikkel Elle Lepperød

A study from the University of Oslo demonstrates that a biologically inspired local learning model, based on Krotov and Hopfield's architecture, inherently produces smooth latent representations with power law covariance spectra, leading to consistently superior adversarial robustness compared to traditional backpropagation-trained networks.

Key insights

• Biologically inspired local learning in the Krotov and Hopfield model naturally produces smooth latent representations with scale-free power law covariance spectra, mirroring characteristics observed in biological systems (e.g., mouse V1) and linked to an optimal balance of accuracy and robustness.
• A direct correlation exists between the smoothness of neural representations (indicated by lower Jacobian norm values and smoother decision boundaries) and adversarial robustness, confirming that geometric properties in hidden representations are crucial for classifier resilience.
• L2 weight regularization and Jacobian regularization exhibit local equivalence in their effects on robustness by promoting smoother representations, explaining their similar performance and shared characteristics in spectral profiles and decision landscapes.

Summary by alphaXiv.