Reproduce “A Tensor-Train Decomposition based Compression of LLMs on Group Vector Systolic Accelerator” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sixiao Huang, Tintin Wang, Ang Li, Ao Shen, Kai Li, Keyao Jiang, Mingqiang Huang, Hao Yu
This paper presents a system that combines tensor-train decomposition for model compression with a specialized FPGA-based accelerator to enable large language model inference on edge devices. The approach reduces first token delay by 1.45x for ChatGLM3-6B and 1.57x for LLaMA2-7B while achieving competitive throughput and network-wide compression ratios up to 1.94x.
Key insights
- Tensor-Train Decomposition (TTD) effectively compresses large weight matrices in LLM linear layers by factorizing them into smaller core tensors, reducing parameter count.
- A specialized Group Vector Systolic Array (GVSA) architecture can efficiently execute the sequential matrix operations inherent to TTD-compressed models.
- Hardware optimizations, including DSP sharing for mixed-precision computation, operator fusion, and optimized data mapping, are critical for maximizing throughput and minimizing data movement on FPGAs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.