Reproduce “Simmering: Sufficient is better than optimal for training neural networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Irina Babayan, Hazhir Aliahmadi, Greg van Anders
Queen's University researchers introduce "simmering," a physics-inspired algorithm that trains neural networks by exploring near-optimal regions of the loss landscape at a finite temperature. This approach consistently improves generalization performance by mitigating overfitting and inherently quantifies prediction uncertainty across classification and regression tasks.
Key insights
- Generalizable neural network representations often reside in near-optimal regions of the loss landscape, particularly in over-parameterized networks with noisy data.
- Introducing a finite temperature generates an entropic force, which reshapes the loss landscape by reducing effective distances along 'sloppy modes' where many parameter combinations yield similar training loss.
- This systematic exploration away from strict loss minima helps mitigate overfitting by deviating from training data idiosyncrasies, leading to more robust and generalizable parameter configurations and intrinsically providing prediction uncertainty.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.