Reproduce “Robust Training under Label Noise by Over-parameterization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sheng Liu, Zhihui Zhu, Qing Qu, Chong You
Sparse Over-Parameterization (SOP), a new training method, directly models and separates label noise within over-parameterized deep neural networks. This approach demonstrated state-of-the-art performance on various benchmarks with competitive training speeds, and is theoretically shown to enable exact noise recovery.
Key insights
- Label noise can be effectively conceptualized and separated as a sparse component within the training process, rather than merely mitigated.
- Specific over-parameterization choices, coupled with tailored gradient descent dynamics, can implicitly enforce desired regularization (L1-sparsity on noise and L2-norm on model parameters), thereby enabling precise separation of clean data from noise.
- The non-convex optimization landscape of SOP is shown to be benign, ensuring that gradient descent reliably converges to global minima that correspond to the exact recovery of true parameters and sparse noise under certain conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.