Reproduce “MemLong: Memory-Augmented Retrieval for Long Text Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weijie Liu, Zecheng Tang, Juntao Li, Kehai Chen, Min Zhang

MemLong, developed by researchers from Soochow University and Harbin Institute of Technology, enhances Large Language Models by efficiently extending their context window through memory-augmented retrieval. It achieves this by storing and retrieving Key-Value (K-V) pairs from frozen lower layers of the LLM, enabling processing of up to 80,000 tokens on a single 3090 GPU while demonstrating superior perplexity on long-context tasks and improved in-context learning performance.

Key insights

  • Efficiently extends LLM context by retrieving and fusing relevant Key-Value (K-V) pairs, enabling processing of up to 80,000 tokens on a single 3090 GPU.
  • Mitigates the 'distribution shift' problem common in memory-augmented models by freezing lower layers of the LLM and using a non-differentiable retriever for stable K-V pair generation.
  • Introduces 'Retrieval Causal Attention' to effectively fuse retrieved chunk-level K-V pairs with local context, allowing for a more direct and efficient integration of external information into the attention mechanism.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.