Reproduce “Demystifying Singular Defects in Large Language Models” on OpenResearch

About this paper

Haoqi Wang, Tong Zhang, Mathieu Salzmann

A theoretical and empirical analysis characterizes the full life cycle of high-norm tokens, or singular defects, in Large Language Models, from their abrupt appearance to their decay. This work extends existing singular defect theory for Vision Transformers to LLMs, revealing causal factors and enabling improvements in quantization and model lineage identification.

Key insights

• The extended singular defect theory accurately explains the distinct life cycle of high-norm tokens in LLMs, including their abrupt explosion (aligned with the layer-wise singular defect direction) and decay (explained by negative eigenvalues of the residual matrix).
• Causal self-attention is identified as a primary architectural factor for the emergence of high-norm tokens, particularly for initial tokens, while certain noninitial tokens exhibit attention-independent high activations.
• High-norm tokens maintain a consistent, stable vector direction across different layers and input texts, which stabilizes early in training and can serve as a unique, data-independent LLM signature.

Summary by alphaXiv.