Reproduce “On 10x Better Scalability: KV Stores Scale Up KV Cache” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiping Yu, Ye Jiarui, He Mengke, Junfeng Liu, Siqiang Luo
SGLANG-LSM introduces a database-inspired system for scalable Key-Value (KV) cache management in Large Language Model (LLM) serving, leveraging Log-Structured Merge-trees (LSM-trees). The system achieved up to 143% higher cache hit rates and reduced Time-to-First-Token (TTFT) latency by up to 24% for 16k prompts compared to existing disk-based methods.
Key insights
- The scalability challenges of LLM KV cache management on disk strongly resemble classic key-value storage problems previously addressed by the database community.
- Applying LSM-tree architectures, especially with key-value separation, effectively mitigates file system bottlenecks by bounding file counts, improving I/O patterns, and converting random tensor writes into sequential appends.
- Dynamically adapting LSM-tree parameters based on workload shifts (e.g., cache-population vs. cache-serving) is crucial for sustained optimal performance in varied LLM inference scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.