Reproduce “Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration” on OpenResearch

About this paper

Peilin Chen, Xiaoxuan Yang

Titanus is a software-hardware co-design solution addressing the Key-Value (KV) cache bottleneck in Large Language Model (LLM) inference by combining an on-the-fly Cascade Pruning-Quantization (CPQ) method with a specialized Computing-in-Memory (CIM) accelerator. The system achieves a 58.9% reduction in KV cache movement, demonstrates 159.9x higher energy efficiency and 49.6x higher throughput compared to an Nvidia A100 GPU, and significantly reduces off-chip memory access by 107.1x due to CPQ, all while maintaining inference accuracy.

Key insights

• A "pruning first, then quantization" sequence in the Cascade Pruning-Quantization (CPQ) method maximizes compression and minimizes overhead by leveraging element-level sparsity.
• The Hierarchical Quantization Extension (HQE) effectively mitigates the Non-independent Per-channel Quantization (NiPCQ) problem in KV cache quantization by dynamically adapting tolerance ranges, drastically reducing re-quantization overhead.
• A co-design approach with specialized hardware units integrated into a Computing-in-Memory architecture can efficiently process dynamic, sparse data on-the-fly with minimal area and power overhead (1.42% area, 0.48% power for specialized units).

Summary by alphaXiv.