Reproduce “Size and depth of monotone neural networks: interpolation and approximation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Mikulincer, Daniel Reichman
The study details the expressive power and efficiency of monotone neural networks with non-negative weights, revealing that monotone ReLU networks cannot universally approximate continuous monotone functions due to convexity. It shows depth-4 monotone threshold networks achieve universal approximation but require an exponential number of neurons for certain monotone functions compared to general networks, a finding established through connections to Boolean circuit complexity.
Key insights
- Monotone neural networks with ReLU activation functions are inherently convex, which fundamentally limits their ability to universally approximate all continuous monotone functions.
- While depth-2 monotone threshold networks are insufficient for general monotone interpolation in dimensions d >= 2, depth-4 monotone threshold networks achieve universal approximation.
- Enforcing monotonicity through non-negative weights can lead to an exponential increase in the number of neurons required for certain monotone functions, compared to unconstrained networks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.