Reproduce “Speeding Up Entmax” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé, Zhenisbek Assylbekov
The paper introduces α-ReLU, a sparse output transformation for neural networks that offers computational speed comparable to optimized softmax. It successfully addresses issues like text degeneration and empty translations in text generation while achieving competitive or superior performance in Neural Machine Translation compared to both softmax and α-entmax.
Key insights
- α-ReLU, despite being significantly less sparse than α-entmax, achieves comparable or superior performance, suggesting that the ability to induce *some* sparsity rather than maximal sparsity is sufficient for beneficial effects.
- The unnormalized outputs of α-ReLU exhibit a strong self-normalization property, where the sum of components concentrates around a consistent value, justifying its practical use in beam search decoding.
- Neural Tangent Kernel (NTK) theory provides a theoretical explanation for the observed similar training dynamics (in terms of steps) across softmax, α-entmax, and α-ReLU models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.