Reproduce “Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers” on OpenResearch

About this paper

Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam

This paper investigates why large language models (LLMs) can be misled into retrieving incorrect facts by irrelevant contextual information, even when prompts are unambiguous. It shows LLMs behave as associative memory models, where contextual cues guide fact retrieval, and develops a synthetic task to mechanistically analyze how transformer components contribute to this behavior.

Key insights

• LLMs' fact retrieval is fragile and susceptible to "context hijacking," suggesting they operate more like associative memory models guided by contextual cues than robust semantic reasoners.
• The value matrix in transformers plays a critical role as an associative memory, functionally mapping aggregated context representations to output tokens in a structured manner.
• Trained embedding matrices can develop a low-rank geometric structure where inner products relate linearly to Hamming distances in a latent concept space, indicating how conceptual associations are encoded.

Summary by alphaXiv.