Reproduce “SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding” on OpenResearch

About this paper

Weihong Xu, Haein Choi, Po-kai Hsu, Shimeng Yu, Tajana Rosing

SLIM introduces an algorithm-hardware co-design for efficient edge inference of sparse large language models, leveraging adaptive thresholding and a heterogeneous DRAM-SSD accelerator. This system delivers 13-18x higher throughput compared to SSD-GPU baselines and up to 10x better energy efficiency than DRAM-GPU systems, while maintaining negligible accuracy loss at 50% sparsity.

Key insights

• Adaptive thresholding for sparsity prediction allows for runtime configurable sparsity, enabling dynamic trade-offs between inference speed and accuracy without requiring costly fine-tuning of the entire LLM.
• Heterogeneous offloading to specialized processing units—DRAM-PIM for latency-critical MHA and sparsity prediction, and SSD-NSP for bulk FFN/MoE weights—effectively mitigates the PCIe bandwidth bottleneck and leverages the cost-effectiveness of high-density NAND flash.
• An algorithm-hardware co-design approach significantly enhances efficiency for edge LLM inference, demonstrating that integrating algorithmic sparsity exploitation with custom memory-centric computing architectures can overcome data movement and resource limitations.

Summary by alphaXiv.