Reproduce “Why do LLMs attend to the first token?” on OpenResearch

About this paper

Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu

Large Language Models learn attention sinks, often directed at the first token, as a fundamental mechanism to prevent information degradation, known as over-mixing and representational collapse, which is crucial for maintaining performance, particularly in processing long contexts. Removing these sinks at inference time severely impacts model utility across various tasks.

Key insights

• Attention sinks function as a learned defense mechanism against over-mixing and representational collapse, preserving distinct token information by controlling the propagation of perturbations within deep Transformer layers.
• Individual attention heads can leverage sinks as "approximate no-ops," directing attention to a low-norm token (like the ` <bos> ` token) to selectively minimize updates when specific semantic conditions are not met.
• The prevalence and strength of attention sinks are causally linked to increased context length and model scale, indicating their growing necessity as models process more information or become deeper.

Summary by alphaXiv.