Reproduce “Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang and 7 more
Lngram v2 adds a lookup branch to Transformers, using discrete codes derived from hidden states to retrieve local n-gram patterns alongside dense computation. Its exact-lookup path uses grouped-query attention and a learned memory, with surrogate gradients for routing during training. On Keye2B, it raises the ten-task average from 47.25 to 47.93 across three seeds, while one tested configuration increases latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.