Reproduce “Understanding KV Residency in Production-Scale Agentic LLM Workloads” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mayank Arya, Agrim Bari, Anjaly Parayil, Yogesh Simmhan, Renee St. Amant, Victor Rühle, Saravan Rajmohan
The authors analyze KV-cache residency across four Microsoft M365 traces containing more than 121 million model calls. They replay policies, including multi-resolution retention (MRR), which stores displaced session state in smaller int4 copies. In a pooled 4.7 TB Research replay, MRR yields a 13.6% context reconstruction rate, versus 31.1% for future-aware eviction; the simulation omits promotion costs and response-quality measurement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.