Reproduce “Learning Activation Functions: A new paradigm for understanding Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohit Goyal, Rajan Goyal, Brejesh Lall
Researchers from UIUC and IIT Delhi introduce Self-Learnable Activation Functions (SLAF), which approximate optimal activation functions as learnable polynomials, providing a mathematical framework for understanding neural network (NN) non-linearities. They theoretically prove SLNNs can approximate any NN with Lipschitz continuous activations and demonstrate improved performance on learning sparse polynomials and competitive results on standard benchmarks, achieving 99.55% test accuracy on MNIST.
Key insights
- SLNNs possess universal approximation capacity, capable of approximating any neural network with Lipschitz continuous activation functions.
- An SLNN can be theoretically represented as a single multivariate polynomial of its input features, where the polynomial degree is the product of the degrees of SLAFs in each layer.
- The inherent parameter redundancy within SLNNs facilitates better optimization and generalization for Stochastic Gradient Descent (SGD) by providing a more navigable loss landscape than direct polynomial regression.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.