Reproduce “Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Wang, Jieyu Li, Yanan Sun, Weifeng He
The Harmonia framework integrates an algorithm-hardware co-design to enable memory- and compute-efficient Large Language Model inference by extending the Block Floating Point format to activations across all LLM layers and aggressively compressing the Key-Value cache. This approach achieves an average 3.08x speedup, 3.84x higher area efficiency, and 2.03x better energy efficiency compared to prior works, alongside up to 66.60% reduction in KV-cache storage.
Key insights
- Model accuracy for BFP activations is highly sensitive to the mantissa bit width but less so to group size, informing the selection of an 8-bit mantissa for general activations to maintain accuracy.
- Attention score distributions reveal that initial and most recent tokens are critical, enabling asymmetric precision allocation in the KV cache (e.g., 8-bit for critical, 4-bit for others) to achieve high compression with minimal accuracy impact.
- Outliers in the Key (K) matrix channels can severely degrade BFP conversion accuracy; a hybrid offline (scaling factors absorbed into weights) and online (incremental channel-wise offsets) smoothing strategy effectively mitigates this issue.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.