Reproduce “An Evolved Universal Transformer Memory” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Edoardo Cetin, Qi Sun, Tianyu Zhao, Yujin Tang

Neural Attention Memory Models (NAMMs), trained with evolutionary algorithms, manage the Key-Value (KV) cache in transformers by intelligently selecting tokens to retain or discard. This approach simultaneously improves transformer performance on long-context tasks, achieving a ten-fold increase on InfiniteBench, while reducing KV cache size by up to 75% and demonstrating zero-shot transferability across models and modalities.

Key insights

  • A learned, rather than heuristic, approach to KV cache management can simultaneously enhance transformer performance and reduce memory footprint, effectively overcoming the traditional trade-off.
  • Evolutionary algorithms are an effective method for optimizing non-differentiable memory management policies, allowing direct improvement of complex, task-specific performance metrics.
  • Memory models that condition solely on attention matrix values can achieve zero-shot transferability across diverse transformer architectures, scales, and input modalities without requiring retraining.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.