Reproduce “Random deep neural networks are biased towards simple functions” on OpenResearch

About this paper

Giacomo De Palma, Bobak Toussi Kiani, Seth Lloyd

Researchers at MIT provided a theoretical proof that random wide deep neural networks with ReLU activation generate binary classifiers inherently biased towards simple functions, meaning they are significantly insensitive to input perturbations and require a non-constant number of changes to alter classification. This work advances the theoretical understanding of deep learning generalization and robustness by rigorously quantifying this 'simplicity bias'.

Key insights

• Random wide deep neural networks with ReLU activation are theoretically proven to generate binary classifiers that are intrinsically 'simple' and insensitive to input changes.
• This simplicity bias of random networks is qualitatively distinct from uniformly random classifiers, which are highly sensitive to single bit flips.
• The absolute output value (classification confidence) correlates linearly with the distance to the classification boundary in both random and trained networks.

Summary by alphaXiv.