Reproduce “Abrupt and spontaneous strategy switches emerge in simple regularised neural networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anika T. Löwe, Léo Touzo, Paul S. Muhle-Karbe, Andrew M. Saxe, Christopher Summerfield, Nicolas W. Schuck
The authors test whether gradual learning can produce insight-like behavior in a perceptual-choice task. Simple networks learn motion and later encounter an unannounced predictive color cue, while learned gates suppress or express feature effects. Online gradient descent produced abrupt, selective, delayed switches in 46 of 99 networks, resembling human behavior, but it addresses behavioral signatures, not awareness or insight mechanisms.
Key insights
- L1-regularization on gate weights is crucial for insight-like behavior, as it strongly suppresses initially irrelevant feature pathways, rendering existing knowledge "silent" until a sudden, noise-induced release.
- Noise in gradient updates, particularly for gate weights, plays a critical role by inducing variability that can trigger the 'un-gating' of suppressed information, leading to abrupt strategy switches.
- The core behavioral signatures of human insight (suddenness, selectivity, and variable delay) can emerge as direct properties of gradual, gradient-based learning within a unified framework, challenging the need for separate complex cognitive modules.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.