Reproduce “QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya, Abhishek Gupta, Sandeep Kumar, Manoj Kumar
QEvict combines selective KV-cache eviction with low-bit storage in a recoverable three-tier hierarchy for long-context decoding. By routing contiguous historical windows among full-precision, INT2, and evicted states, it improves future-context coverage and maintains stronger LongBench, RULER, and GSM8K performance than matched-memory eviction baselines, while current implementations still incur substantial FlashAttention-2 overhead.
Key insights
- Attention importance is persistent but not fixed: windows that are temporarily inactive can later regain relevance, making recoverability useful for long-context decoding.
- Window-level routing reduces selection churn and preserves coherent context better than independent token decisions; in diagnostics, FMM fell from 51.73% to 46.38% and churn from 0.0170 to 0.0012.
- At matched memory, QEvict can retain historical information across precision levels: binary eviction permanently discarded 64.8% of attention mass, compared with 35.7% for QEvict, with INT2 scores showing 0.9824 cosine agreement with FullKV rankings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.