Reproduce “HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models” on OpenResearch

About this paper

Jack Goffinet, Casey Hanks, David E. Carlson

Associative Memory HiPPO adds an addressable polynomial memory bank to HiPPO’s fixed-size summary of input history, enabling explicit key–value writes and reads. On a synthetic associative-recall task, it reaches 100.0% test accuracy, versus 33.9% for a parameter-matched single-layer Transformer. A two-layer Transformer reaches 99.7% under different conditions, and HiPPO variants underperform modern state space models on character-level WikiText-2.

Key insights

• Advanced memory capabilities characteristic of modern, high-performing SSMs can be achieved through explicit and interpretable modifications within the HiPPO framework, challenging the notion that complex implicit dynamics are always necessary.
• Explicitly designed memory mechanisms, such as salience signals for adaptive weighting or continuous address spaces for associative recall, offer direct observability and analyzability of how historical information is processed and utilized.
• While explicit interpretability often entails a performance trade-off on general-purpose tasks like language modeling, these task-aligned explicit mechanisms can significantly outperform opaque baselines on synthetic tasks designed to test specific memory functionalities.

Summary by alphaXiv.