Reproduce “A dynamic parallel method for performance optimization on hybrid CPUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luo Yu, Liu Yucheng, Shen Haihao
Researchers at Intel Corporation developed a dynamic parallel method to optimize large language model inference on hybrid CPUs, which adaptively balances workloads across heterogeneous cores. The method achieved up to a 3.7x speedup for Llama2-7B inference compared to `llama.cpp` and boosted kernel performance by up to 85% for compute-intensive tasks, while exceeding 90% memory bandwidth utilization for memory-bound operations.
Key insights
- Dynamically adjusting workload distribution based on observed core performance is crucial for unlocking the full potential of heterogeneous hybrid CPUs for LLM inference.
- Performance ratios are not static; they must adapt in real-time to system conditions and different workload characteristics (e.g., prefill vs. decode phases).
- Even in memory-bound phases, optimized dynamic scheduling can push memory bandwidth utilization close to theoretical limits (over 90%), significantly improving decode speeds.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.