Reproduce “Overview frequency principle/spectral bias in deep learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo
Researchers from Shanghai Jiao Tong University and Shanghai AI Lab consolidate evidence for the Frequency Principle, demonstrating that deep neural networks implicitly learn low-frequency functions before high-frequency ones. This work provides a fundamental explanation for DNN generalization, consistently observed across diverse architectures and datasets, and offers practical implications for deep learning design.
Key insights
- Common activation functions' inherent smoothness dictates a frequency-dependent decay rate of loss function gradients, causing faster convergence for low-frequency components.
- In the infinite-width (NTK) limit, lower-frequency eigenvectors of the Neural Tangent Kernel correspond to larger eigenvalues, mathematically proving faster learning rates for these modes.
- The F-Principle unifies explanations for various deep learning phenomena, including the generalization puzzle, the efficacy of early stopping, and the challenges networks face with intrinsically high-frequency tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.