Reproduce “Mugi: Value Level Parallelism For Efficient LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Price, Prabhu Vellaisamy, John Shen, Di Wu
Mugi is a hardware architecture that generalizes Value Level Parallelism (VLP) for Large Language Model (LLM) inference, efficiently handling both complex nonlinear operations and asymmetric, small-batch General Matrix Multiply (GEMM) computations. It achieves up to 2.07x higher throughput and 3.11x higher energy efficiency for end-to-end Llama 2 70B workloads, while also reducing operational and embodied carbon emissions by approximately 1.45x and 1.48x respectively.
Key insights
- Value Level Parallelism can be effectively extended beyond traditional GEMM to perform accurate and efficient nonlinear function approximations by leveraging input value distributions and temporal subscription.
- Optimizing VLP for LLM-specific characteristics like asymmetric quantization and small-batch inference requires a novel data mapping within the VLP array and careful buffer minimization techniques.
- A unified hardware architecture that reuses a single compute array for both GEMM and nonlinear operations can significantly reduce area overhead, power consumption, and consequently, both operational and embodied carbon emissions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.