Reproduce “Learning Agent Execution for KV-Cache Management in Agentic Serving” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rui Zhang, Chaeeun Kim, Shaoting Feng, Kuntai Du, Yuhan Liu, Yi Zhong, Cheng-Wei Ching, Junchen Jiang and 1 more

CacheScout learns transitions between agents to retain reusable anchors—system, tool, skill, and example prefixes—in the KV cache and can prefetch anchors. Across workloads, fixed context comprises 53–62% of prompt tokens. Predictive eviction improves hit rate by 18–22 points; on GAIA, prefetch lowers latency from 347 ms to 251 ms, but gains depend on predictable routing and add GPU work.

Key insights

  • Dynamic agent execution patterns in multi-agent LLM systems can be effectively captured and predicted online using lightweight Markov chain models with minimal overhead.
  • Proactively retaining KV-cache blocks based on learned agent execution semantics and predicted future reuse significantly improves cache hit rates and reduces inference latency compared to reactive policies.
  • Blending semantic-aware survival scores with traditional recency signals and reconstruction costs allows for robust and efficient KV-cache eviction decisions that adapt to both agent predictability and memory pressure.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.