Reproduce “RR-Evict: Fine-Grained Prefix Cache Eviction beyond LRU for Agentic LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaifeng Pan, Chris Wu, Zhengding Hu, Xinwei Qiang, Zhongkai Yu, Yufei Ding
RR-EVICT spreads eviction across idle agent histories by removing tail chunks in turn, preserving partial prefixes when active contexts fit in GPU memory but all live idle histories do not. On τ²-bench, it reduces P99 TTFT by up to 75.4% and P99 uncached prompt tokens by up to 65.7% versus LRU, maximum reductions across tested concurrency levels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.