Reproduce “Deriving Activation Functions Using Integration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Allen Hao Huang, Imanol Schlag
Researchers at EPFL and ETH Zürich introduce a "gradient-first" methodology for designing activation functions, deriving novel functions like xIELU and xIPReLU by integrating desired gradient properties. These functions consistently achieve lower perplexity and loss compared to state-of-the-art ReLU² and SwiGLU on 1.1B and 3B Llama models, demonstrating adaptive nonlinearity through trainable parameters.
Key insights
- The behavior of an activation function's gradient, particularly its ability to provide trainable negative-valued gradients for negative inputs and linearly increasing positive gradients, is critical for model performance and training stability.
- Trainable parameters, when designed to control essential gradient characteristics, can significantly enhance activation function performance and allow for adaptive non-linearity within deep networks.
- Deep neural networks, especially LLMs, benefit from adaptive non-linearity, where the effective strength of the nonlinearity (controlled by trainable parameters) can decrease in deeper layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.