Reproduce “Coding schemes in neural networks learning classification tasks” on OpenResearch

About this paper

Alexander van Meegen, Haim Sompolinsky

Researchers at Harvard University developed an exact mean-field theory to explain how emergent "coding schemes" (analog, redundant, sparse) arise in wide, feature-learning neural networks, demonstrating that neuronal nonlinearities and spontaneous symmetry breaking dictate these representations. They found that improved generalization in these networks primarily stems from a reduction in predictor variance.

Key insights

• Different neuronal nonlinearities lead to qualitatively distinct emergent coding schemes: linear networks exhibit analog coding, sigmoidal networks show redundant coding, and ReLU networks develop sparse coding.
• Spontaneous permutation symmetry breaking in the weight posterior is a crucial mechanism for the emergence and stability of redundant and sparse coding schemes in nonlinear networks.
• The improved generalization performance of non-lazy networks, compared to the lazy regime, is primarily driven by a reduction in the predictor variance rather than a significant alteration of the mean predictor.

Summary by alphaXiv.