Reproduce “Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data” on OpenResearch

About this paper

Nikita Tsoy, Nikola Konstantinov

Researchers from INSAIT, Sofia University, theoretically characterize simplicity bias in two-layer neural networks for general, non-linearly separable datasets, showing that features initially condense around a data-dependent function and this bias intensifies in later training stages for XOR-like data. Their findings suggest that models might "forget" complex features useful for out-of-distribution generalization, a phenomenon empirically observed with decreased OOD accuracy on image datasets during later training phases.

Key insights

• In early training stages, a "simplicity bias" causes prominent neurons to condense around global extrema directions of a data-dependent function G, independent of the number of hidden neurons.
• Simplicity bias propagates as network weights grow and, for non-linearly separable XOR-like data, can intensify significantly in the infinite training time limit, leading to reliance on a minimal set of "simple" features.
• The pursuit of minimal training loss in later stages can exacerbate simplicity bias, potentially causing networks to "forget" complex features that are beneficial for out-of-distribution generalization.

Summary by alphaXiv.