Reproduce “Vorn: Residual Direction, Familial Eviction, and the Granularity Rescue Spectrum” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
L. Penney
A study demonstrates that the efficacy of KV-cache eviction methods is conditional on the specific large language model family, introducing "vorn," a residual-direction based eviction strategy that proved effective for most models while others favored attention-weight methods. This work also revealed that using sentence-level retention consistently lowers the cost per correct answer by 1.25x-4.03x and that vorn methods generally require less peak GPU memory.
Key insights
- The efficacy of KV-cache eviction methods is family-conditional, challenging the implicit assumption of universal effectiveness and indicating that optimal strategies are model-specific.
- Retention granularity significantly impacts performance and cost, with sentence-level grouping consistently reducing the cost per correct answer and demonstrating a "granularity rescue spectrum" across model families.
- Targeted falsification experiments revealed that neither attention-pattern architecture nor the originating laboratory identity solely explain the observed channel favoritism, pointing to more specific architectural or training variables.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.