Reproduce “Activation functions are not needed: the ratio net” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chi-Chun Zhou, Hai-Long Tu, Yue-Jie Hou, Zhen Ling, Yi Liu, Jian Hua
The 'ratio net' architecture is presented, which replaces conventional activation functions with a fractional polynomial for nonlinearity in deep neural networks. This approach consistently yields improved classification accuracy, faster convergence, and reduced parameter counts compared to traditional Multi-Layer Perceptrons and Radial Basis Function networks across benchmark datasets.
Key insights
- Nonlinearity in neural networks can be effectively introduced using fractional polynomial forms, providing a new mechanism distinct from conventional activation functions.
- The ratio net's universal approximation capabilities are rooted in the theory of Padé approximants, extending their function approximation power to multi-dimensional inputs and outputs.
- Adopting fractional forms for nonlinearity can lead to simpler and more stable neural network architectures, bypassing common issues related to activation function selection and initialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.