Reproduce “Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Cendekia Airlangga, Hilal AlQuabeh, Munachiso S Nwadike, Kentaro Inui
This study delves into the memory dynamics of Mamba, a state-space language model, by mechanistically explaining the co-emergence of primacy and recency effects. It identifies specific recurrent channels acting as long-term memory for initial inputs and attributes recency to the model's exponential decay, demonstrating how the dynamic discretization parameter (∆t) adapts to input regularity to modulate information retention.
Key insights
- Mamba exhibits a consistent U-shaped recall accuracy profile (primacy and recency), with the degree of accuracy in central positions significantly influenced by the semantic regularity of input sequences.
- A sparse, localized subset of channels within Mamba's selective SSM blocks function as specialized 'long-term memory' units, causally responsible for persistently encoding early inputs, thereby enabling the primacy effect.
- Recency is linked to Mamba's inherent exponential decay in state updates, where recent inputs undergo fewer decay steps, and the dynamic ∆t parameter adaptively modulates memory retention and forgetting based on input frequency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.