Reproduce “Inference Performance Optimization for Large Language Models on CPUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui and 2 more
An end-to-end optimization framework, xFasterTransformer, was developed by Intel Corporation to enhance Large Language Model inference on CPUs, introducing techniques like SlimAttention and INT8 KV caching. This solution demonstrated a 2.85X performance gain for Llama2-70B in distributed inference and achieved superior latency for SlimAttention compared to FlashAttention on CPU architectures.
Key insights
- CPU-specific attention mechanisms can outperform GPU-optimized counterparts on CPU architectures, challenging the assumption that only GPU-centric solutions are optimal.
- Fine-grained INT8 quantization of the Key-Value cache, coupled with custom hybrid data type kernels, significantly reduces memory footprint while preserving inference precision.
- Distributed inference optimizations, including smart data broadcasting and zero-copy techniques, are critical for achieving scalable and low-latency LLM performance on multi-socket CPU systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.