Reproduce “Improving the Efficiency of Transformers for Resource-Constrained Devices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hamid Tabani, Ajay Balasubramaniam, Shabbir Marzban, Elahe Arani, Bahram Zonooz
A parameter clustering technique was developed by researchers at NavInfo Europe to enhance the inference efficiency of Transformer models for resource-constrained devices. This approach yielded up to 38% speedup and 39% energy reduction, incurring a minimal accuracy loss, such as 0.1% Top-1 accuracy for the DeiT model using 64 clusters.
Key insights
- Matrix multiplication layers in Transformers account for over 50% of execution time and over 40% of memory usage, making them critical targets for efficiency improvements.
- Parameter clustering effectively compresses model parameters, leading to a nearly 4x reduction in memory footprint and bandwidth requirements with negligible overhead from centroid tables.
- The method offers a tunable trade-off between accuracy and efficiency, allowing customization based on application needs, with per-layer clustering providing better accuracy for fewer clusters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.