Reproduce “Understanding Deep Neural Networks with Rectified Linear Units” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Raman Arora, Amitabh Basu, Poorya Mianjy, Anirbit Mukherjee
This study offers a theoretical framework for deep neural networks using ReLU activations, demonstrating their expressive equivalence to continuous piecewise linear functions and quantifying the advantages of depth through gap theorems. Researchers also present an algorithm for globally optimal training of two-layer ReLU DNNs, characterized by polynomial time complexity in data size for fixed input dimensions and width.
Key insights
- ReLU DNNs precisely represent the class of continuous piecewise linear functions, providing a fundamental characterization of their expressive power.
- Network depth offers substantial computational efficiency, enabling deep ReLU DNNs to represent complex functions with significantly fewer parameters than shallower alternatives, even for smooth function families.
- Globally optimal training for 2-layer ReLU DNNs is achievable, with a theoretical complexity polynomial in data size for fixed input dimension and width, despite exponential dependence on other factors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.