Reproduce “Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiwon Song, Dongwon Jo, Yulhwa Kim, Jae-Joon Kim
A training-free KV cache compression method, Reasoning Path Compression (RPC), enhances the efficiency of large language models for complex reasoning tasks by exploiting semantic sparsity in generated trajectories. RPC achieves up to a 1.68x increase in inference throughput and over 50% reduction in peak memory usage, enabling models to generate extremely long reasoning paths while preserving high accuracy.
Key insights
- LLM-generated reasoning paths contain 'semantic sparsity,' characterized by redundant steps, re-evaluations, or paraphrased duplications that can be safely removed.
- Dynamically evaluating token importance using an attention-based score, derived from a small 'selector window' of recent tokens, effectively preserves the coherence and accuracy of the reasoning process.
- Periodic KV cache compression, coupled with dynamic reassessment of all relevant tokens, provides an efficient mechanism to manage cache growth without significant accuracy degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.