Reproduce “SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahuan Yu, Mingtao Hu, Zichao Lin, Minjia Zhang
An LLM inference system, SuperInfer, integrates an SLO-aware rotary scheduler and a high-performance KV cache memory manager to leverage Superchip architectures. It overcomes NVLink-C2C underutilization and achieves up to 74.7% higher Time-To-First-Token SLO attainment, enabling responsive LLM serving under high request rates.
Key insights
- Existing LLM serving software on Superchips underutilizes NVLink-C2C bandwidth due to fragmented, small KV cache transfers and high kernel launch overhead, not hardware limitations.
- Proactive, OS-inspired rotary scheduling, guided by a Virtual Lag Time (VLT) metric, effectively balances TTFT and TBT SLOs by dynamically prioritizing requests and preempting those with less urgency.
- Eliminating data races via eager block rotation and consolidating small KV cache segments into larger, contiguous blocks are crucial for achieving near-ideal full-duplex NVLink-C2C transfer efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.