Reproduce “Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang and 13 more

The paper introduces conditional memory, instantiated as the Engram module, as a new axis of sparsity for Large Language Models to efficiently retrieve static knowledge through scalable lookup operations. This approach outperforms traditional Mixture-of-Experts models across diverse benchmarks, including knowledge-intensive tasks and long-context understanding, while enabling massive parameter scaling with minimal computational overhead.

Key insights

  • Linguistic signals exhibit a duality between compositional reasoning and static knowledge retrieval, necessitating distinct and complementary architectural solutions.
  • An optimal balance exists in allocating sparse parameters between conditional computation (MoE experts) and conditional memory (Engram modules), with a hybrid approach outperforming pure MoE.
  • Integrating conditional memory accelerates feature composition, effectively 'deepening' the model by enabling earlier prediction convergence in shallower layers.
  • The deterministic nature of Engram's retrieval allows decoupling parameter storage from computation, facilitating massive memory scaling with minimal inference overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.