Reproduce “AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ilias Bournias, Lukas Cavigelli, Georgios Zacharopoulos
AcceLLM accelerates Large Language Model inference in cloud environments by employing redundant Key-Value cache copies, enabling dynamic instance role switching and superior load balancing. This approach leads to up to a 30% reduction in Job Completion Time and a nearly 30% increase in throughput per instance compared to state-of-the-art baselines.
Key insights
- Maintaining redundant Key-Value (KV) cache copies across instances allows dynamic switching between prefill and decoding roles without costly full cache transfers.
- Strategic KV cache replication facilitates effective load balancing during decoding, ensuring even distribution of batched requests across instances and maximizing GPU utilization.
- Transferring KV caches per layer during both prefill and decoding, rather than at the end of a phase, efficiently utilizes high-speed interconnects and minimizes Job Completion Time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.