Reproduce “A Method for Building Large Language Models with Predefined KV Cache Capacity” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhonghua Yi, Ge Niu, Lei Wang, Wei Tang, Liqiu Zhang

The Bounded-Cache Transformer (BCT) method enables large language models to operate with a predefined, fixed Key-Value (KV) cache capacity during inference. This approach maintains constant memory usage and stable inference speeds after exceeding 2048 tokens, while achieving BLEU scores comparable to traditional methods (within 3 points difference).

Key insights

  • A hard-bounded KV cache, combined with a dynamic additive update rule, can effectively prevent memory explosion in Transformers without significant performance degradation.
  • The BCT approach functions as a hybrid, merging the bounded memory characteristics of Recurrent Neural Networks (RNNs) with the powerful self-attention mechanism of Transformers.
  • By continuously refining a fixed-size memory based on incoming tokens, the model adaptively retains relevant long-range contextual information within its capacity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.