Reproduce “Transformers are Multi-State RNNs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, Roy Schwartz

TOVA evicts cached tokens with the lowest current-query attention score, bounding a decoder transformer’s key-value state without retraining. It treats the growing cache as a recurrent state and averages attention across heads for eviction. On PG-19, LLaMA-2-7B reaches perplexity 7.41 with 512 states versus 7.16 with 4,096; results are near full-cache performance on this task, but retrieval and story tasks need larger caches.

Key insights

  • Decoder-only Transformers can be conceptually understood as Multi-State RNNs (MSRNNs), where the KV cache functions as a dynamically growing state matrix, bridging the gap between these architectures.
  • Effective KV cache compression is achievable without re-training pre-trained LLMs by leveraging attention scores to identify and retain important tokens.
  • Analysis of retained tokens reveals that while recency matters, specific elements like the first token of a sequence, punctuation, and certain grammatical markers are disproportionately important for long-range context.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.