Reproduce “FlatCake: A Flat KVCache Storage Architecture for LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang and 2 more
FlatCake places language-model KV caches across DRAM and SSD for reuse of long histories. It assigns immutable blocks at creation, groups session data in layer-major order, and restores SSD data through coalesced reads to GPU memory. On a Claude Code replay with DeepSeek-V4-Flash at 1.0 QPS, it records 7.57-second mean TTFT and 98.3K tokens/s, versus Mooncake’s 12.24 seconds and 61.6K tokens/s.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.