Reproduce “Learned Token Pruning for Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, Kurt Keutzer
Researchers from UC Berkeley and Samsung Semiconductor developed Learned Token Pruning (LTP), an adaptive, threshold-based token pruning method for Transformers. This approach reduces FLOPs by up to 2.10x and achieves real-world speedups of up to 2.0x on commodity hardware with less than a 1% accuracy drop, while outperforming existing pruning methods and demonstrating robustness to varying sequence lengths.
Key insights
- Threshold-based pruning is significantly more computationally efficient than 'top-k' selection for token removal, enabling speedups on standard hardware without specialization.
- Learning optimal pruning thresholds automatically through a differentiable soft mask allows for adaptive, content-aware pruning that dynamically adjusts to each input sequence.
- A training regimen involving soft pruning followed by hard binarization and fine-tuning effectively balances the optimization of pruning strategy with maintaining model accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.