Reproduce “Efficient In-Memory Acceleration of Sparse Block Diagonal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
João Paulo Cardoso de Lima, Marc Dietrich, Jeronimo Castrillon, Asif Ali Khan
An automated framework from TU Dresden and ScaDS.AI enables efficient deployment of structured sparse large language models on analog Compute-in-Memory (CIM) accelerators. This approach reduces the required CIM arrays by up to 87%, decreases inference latency by up to 1.74x, and lowers energy consumption by up to 1.74x compared to dense CIM baselines.
Key insights
- Achieving full efficiency from combining structured sparsity (specifically Monarch factorization) with analog CIM requires a co-design approach for mapping and scheduling, rather than a naive application.
- Capacity-optimized mapping (DenseMap) dramatically improves CIM array utilization (to 78.8% from 20.4% in SparseMap) and reduces the total number of required CIM arrays by up to 87%, making resource-constrained deployments more viable.
- The optimal strategy for sparse LLM acceleration on CIM depends on system-level constraints, especially the availability and precision of ADCs, influencing the trade-offs between parallelism and array utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.