Reproduce “APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos

APEX is a CPU–GPU scheduling system for online LLM inference when long KV caches exceed available GPU memory. By keeping linear layers in a unified GPU batch while asynchronously executing selected attention operations on the CPU and delaying synchronization, it improves throughput over NEO by up to 72% and over vLLM by up to 96% on tested T4 workloads, while maintaining comparable or lower per-token latency.

Key insights

  • Avoiding duplicated GPU linear work is central: asynchronous overlap can be more effective than asymmetric pipelining when CPU attention is much slower than GPU attention.
  • Longer generated outputs favor APEX because decode-dominated workloads provide more opportunity to hide CPU attention behind GPU execution; gains plateau once CPU throughput becomes the limiting factor.
  • Scheduling and synchronization policy materially affect hybrid inference performance, but the implementation depends on hardware/model profiling and sufficient CPU-side batch size to amortize runtime overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.