Reproduce “TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents” on OpenResearch

About this paper

Abhishek Vijaya Kumar, Bhaskar Kataria, Byungsoo Oh, Emaad Manzoor, Rachee Singh

TVCACHE, a stateful tool-value cache developed at Cornell University, enhances the efficiency of Large Language Model (LLM) agent reinforcement learning post-training by reusing the results of external tool calls. The system achieves up to a 6.9 times reduction in median per-tool-call execution time and high cache hit rates (e.g., 76.2% for EgoSchema) across diverse workloads, without degrading the agent's learning performance.

Key insights

• Correctly caching stateful tool calls requires a robust mechanism like the TCG and LPM to guarantee that the environmental state is identical upon cache reuse, which is crucial for maintaining the integrity of RL training.
• Optimizing cache efficiency necessitates strategic management of sandbox states, including selective snapshotting to prioritize expensive operations and efficient forking (proactive and reactive) to minimize overhead during cache misses.
• The applicability and hit rates of a stateful cache can be further improved for stateless tools by performing LPM solely on state-modifying tool subsequences, enhancing reuse opportunities without compromising correctness.

Summary by alphaXiv.