Reproduce “MELODI: Exploring Memory Compression for Long Contexts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz, Jesper Andersen
Google DeepMind researchers developed MELODI, a hierarchical memory compression architecture for transformer models that efficiently processes very long documents using fixed-size context windows. It achieved lower perplexity on long-context benchmarks while reducing memory consumption by up to 8 times compared to strong baselines like the Memorizing Transformer.
Key insights
- A hierarchical memory approach, with distinct recurrent short-term and compressed long-term components, is effective for managing information across extended sequences.
- Short-term memory primarily handles immediate context, while long-term memory is critical for retaining crucial information from distant history, enhancing overall recall.
- Significant memory efficiency gains are achieved by compressing key-value pairs from context tokens for long-term storage, rather than storing them directly.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.