Reproduce “Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jongjin Baek, Won Ji, Seungjae Yoo, Joo-Young Kim
The authors propose a hot–cold hierarchy for agent sessions: active KV cache stays in HBM, while paused cache moves to HBF. In simulation at 48 sessions, it achieves 14 ms TBT per agent, below a 50 ms SLO, and raises capacity from 165 to 3,900 sessions. The results support HBF as a cold-tier complement to HBM under these conditions, not a replacement.
Key insights
- The evaluated workload has strongly bimodal locality: at 48 active sessions, the hot set occupies about 96 GiB and accounts for roughly 98% of KV reads while representing only around 3% of resident KV capacity.
- High-capacity storage alone is insufficient for agent serving; active decoding must retain an HBM-resident working set to sustain bandwidth and meet token-latency targets.
- Because KV states are immutable after generation, writing blocks only on eviction substantially reduces flash traffic and improves endurance without requiring repeated updates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.