Reproduce “CacheFlow: Efficient LLM Serving via Automated 3D-Parallel KV Cache Restoration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sean Nian, Hanzhang Shen, Zhiyu Wu, Jiahao Fang, Qilong Feng, Fan Lai

CacheFlow restores evicted key/value caches by coordinating recomputation and loading across token chunks, layer blocks, requests, and model shards. Its batch-aware scheduler varies the recompute/load boundary, while cached boundary activations let shards restore concurrently. In Qwen3.5-9B OSWorld 2.0 input-length tests at 10 Gbps, mean TTFT speedups were 3.00× over Cake, 3.96× over LMCache, and 4.82× over vLLM.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.