Reproduce “Theory of the Frequency Principle for General Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Luo, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang
Researchers from Purdue and NYU Abu Dhabi developed a general theoretical framework establishing the Frequency Principle (F-Principle) for deep neural networks. Their work rigorously demonstrates that DNNs prioritize learning low-frequency components across all training stages Dinitial, intermediate, and final Dfor diverse architectures, activation functions, data distributions, and loss functions.
Key insights
- The Frequency Principle is a universal phenomenon in deep neural networks, consistently observed across diverse architectures, activation functions, data, and loss functions.
- The regularity (smoothness) of the network's function space, parameterized by `k`, directly dictates the decay rate (`eta^-m`) of high-frequency components of the loss function.
- The F-Principle not only holds at the initial training stage but also persists through intermediate training periods and as the network converges to a global minimizer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.