Reproduce “Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hamidreza Alikhani, Nazmul Takbir, Matin Khajavi, Rajat Bhattacharjya, Sangeetha Abdu Jyothi, Nikil Dutt
Fairway is an admission scheduler for prefill queues that balances cached-prefix reuse against delays to older requests. It ranks requests by reuse and relative wait, then uses an arrival-order lane and work charges to limit overtaking within each scheduling step. In an 11-scenario evaluation on one GPU, its p99 TTFT beat the best non-starving baseline in 10 scenarios by 2–18% and tied in one; results do not establish performance beyond the tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.