Reproduce “Augmenting Self-attention with Persistent Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, Armand Joulin
The all-attention layer replaces a Transformer’s separate feedforward sublayer by adding learned, context-independent key–value vectors to the self-attention pool. Each operation can therefore use current context and persistent learned information together. On WikiText-103, it reports 20.6 test perplexity with 133M parameters, versus 24.0 with 151M for Transformer-XL Standard. This comparison shows better measured language-modeling performance with fewer parameters on this benchmark, not faster execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.