Reproduce “Needle in the Haystack for Memory Based Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, Soham Dan

IBM Research presents Larimar, an encoder-decoder LLM augmented with an external, dynamically updatable associative memory, demonstrating robust recall of specific information within contexts up to 1 million tokens. This architecture achieves 100% recall on passkey tests at 1M tokens and significantly outperforms larger baseline models like Mistral 7B and Phi-3-Mini on needle-in-the-haystack tasks at 137K tokens, all without requiring long-context-specific training.

Key insights

  • Dynamically adaptable external associative memory can effectively address long-context information retrieval, decoupling memory footprint from GPU VRAM.
  • Prefix-based key generation is crucial for accurate memory retrieval in memory-augmented LLMs.
  • Generalization to vastly longer contexts than training data is possible without task-specific fine-tuning when an efficient external memory mechanism is employed.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.