Reproduce “SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyunwoo Kim, Byoungchan Ko, Minseok Kang, Minwoo Kim, Dongjin Lee, Jaehoon Lee, Sungroh Yoon, Dahuin Jung
SSDi8 introduces a post-training quantization framework for Mamba-2's Structured State Space Duality (SSD) architecture, establishing a persistent INT8 execution path within the SSD layers. This approach achieves inference speedups of up to 1.47x over FP16 and 1.38x over prior quantization methods, while maintaining accuracy comparable to full-precision models across various benchmarks and exhibiting improved long-context stability.
Key insights
- The distinctive computational patterns within Mamba-2's SSD layers, including heterogeneous dimensional partitioning and frequent intertwined operations, demand tailored quantization strategies for accuracy preservation.
- Strategic reformulation of element-wise operations can enable full INT8 execution paths in complex architectures without substantial accuracy loss, especially when leveraging input sparsity.
- A per-channel mean correction applied at the output projection layer effectively counters accumulated quantization errors across SSD layers, improving overall model accuracy with minimal additional computational overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.