Reproduce “SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference” on OpenResearch

About this paper

Jiho Shin, Hoeseok Yang, Youngmin Yi

SparseInfer presents a training-free method to predict activation sparsity in ReLU-fied Large Language Models (LLMs), which significantly accelerates inference. The approach achieves up to 1.79x speedup over a dense baseline and 1.30x over a state-of-the-art training-based predictor while reducing predictor memory usage by 4.38x and predictor latency by 3.66x, all with negligible accuracy degradation.

Key insights

• Activation sparsity can be effectively predicted by simple sign-bit comparisons (XOR operations) of inputs and weights, eliminating the need for complex, training-based neural network predictors.
• Adaptive adjustment of prediction conservativeness via a tunable parameter (α) is crucial for maintaining accuracy, especially in early layers where activation distributions may vary, allowing for flexible performance-accuracy trade-offs.
• An efficient CUDA implementation, including pre-fetching, parallel bitwise operations, and skipping computations based on predicted sparsity, is essential for realizing substantial end-to-end inference acceleration.

Summary by alphaXiv.