Reproduce “ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition” on OpenResearch

About this paper

Keran Zheng, Yinting Huang, Zhewen Yu, Christos-Savvas Bouganis

ITERA-LLM, developed at Imperial College London, is a software-hardware co-design framework that boosts sub-8-bit Large Language Model inference on FPGAs using iterative tensor decomposition and quantization. The framework demonstrates up to 4.9% BLEU score improvement and 12.1% to 41.1% latency reduction in linear layers compared to quantization-only methods.

Key insights

• Iterative application of SVD to residuals after initial decomposition and quantization can effectively mitigate errors, enabling higher accuracy at sub-8-bit precision.
• Dynamically allocating decomposition ranks based on individual layer sensitivity is crucial for maximizing the overall model accuracy under a fixed compression budget.
• A co-design approach, considering both algorithmic compression and hardware constraints through analytical models, is essential for finding Pareto-optimal solutions for LLM deployment on FPGAs.

Summary by alphaXiv.