Reproduce “Linking In-context Learning in Transformers to Human Episodic Memory” on OpenResearch

About this paper

Li Ji-An, Corey Y. Zhou, Marcus K. Benna, Marcelo G. Mattar

Research from UCSD and NYU demonstrates that 'induction heads' in Transformer models, critical for in-context learning, exhibit behavioral and mechanistic similarities to the Contextual Maintenance and Retrieval (CMR) model of human episodic memory. This work quantifies these parallels using a novel CMR distance metric, showing that CMR-like heads emerge during training and are causally linked to in-context learning performance.

Key insights

• Induction heads in Transformers exhibit attention patterns, such as temporal contiguity and forward asymmetry, that qualitatively mirror human episodic memory recall biases.
• The Contextual Maintenance and Retrieval (CMR) model of human memory can be formally mapped to a specific computational structure within a Transformer's induction head, specifically the Q-composition type.
• The development of human-like memory biases in LLMs during training suggests these biases may reflect normative principles for effective memory recall, rather than mere epiphenomena.

Summary by alphaXiv.