Reproduce “Practical Online KV Cache Compaction for LLM Agents: An Empirical Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yujian Liu, Jiabao Ji, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang
Researchers from UC Santa Barbara and LinkedIn empirically investigated practical online Key-Value (KV) cache compaction for Large Language Model (LLM) agents, evaluating how different proxy query sources influence task performance and inference efficiency when adapting Token Eviction and Attention Matching methods. The study demonstrated that online compaction significantly reduces KV memory footprint and increases serving throughput while preserving most of the task performance on larger models.
Key insights
- Delaying compaction to leverage future-turn queries from the agent's subsequent generations consistently recovers much of the task performance degraded by immediate compaction.
- The simpler Token Eviction (TE) method demonstrated robustness and competitiveness with the more complex Attention Matching (AM), suggesting AM's optimizations offer limited benefit with imperfect online proxies.
- Compaction influences agent behavior, leading to longer trajectories and increased repetition of search queries, indicating agents adapt to weakened internal context by interacting more with the environment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.