Reproduce “Understanding KV Residency in Production-Scale Agentic LLM Workloads” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mayank Arya, Agrim Bari, Anjaly Parayil, Yogesh Simmhan, Renee St. Amant, Victor Rühle, Saravan Rajmohan

The authors analyze KV-cache residency across four Microsoft M365 traces containing more than 121 million model calls. They replay policies, including multi-resolution retention (MRR), which stores displaced session state in smaller int4 copies. In a pooled 4.7 TB Research replay, MRR yields a 13.6% context reconstruction rate, versus 31.1% for future-aware eviction; the simulation omits promotion costs and response-quality measurement.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.