Reproduce “Asymmetric Polynomial-Linear Units” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Mouldon
APoLU extends Transformer feed-forward activations with learned positive polynomial terms, a smooth negative branch, and coefficients combining shared values with per-channel deviations. It caps the positive contribution to retain a monotonically increasing, asymptotically linear response. On FineWeb-Edu, it achieves the lowest validation loss among tested activations at 49.4M and 235.3M parameters; the larger-scale result uses one run per activation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.