Reproduce “Capturing AI's Attention: Physics of Repetition, Hallucination, Bias and Beyond” on OpenResearch

About this paper

Frank Yingjie Huo, Neil F. Johnson

Researchers from George Washington University developed a first-principles physics theory that models the Attention mechanism of large language models as a system of interacting "spins" governed by a Hamiltonian, explaining the emergence of behaviors such as repetition, hallucination, and bias. This framework treats common AI challenges as manifestations of underlying physical phenomena, offering a new lens for analysis and mitigation.

Key insights

• The Attention mechanism can be understood as a physical system with interacting "spins" (tokens) and a "bath" (vocabulary), governed by a Hamiltonian derived from query-key dot products.
• AI pathologies like hallucination and harmful content generation are explained as projections of the Context Vector onto token embeddings that cross specific boundaries in the embedding space.
• Bias is modeled as a linear perturbation to token embeddings that modifies the effective Hamiltonian, thereby rotating or shifting the boundaries between different types of model outputs.
• Repetition in LLMs arises from attractor-like dynamics in the model's parameter space, where the Context Vector repeatedly converges to the same token, analogous to physical systems settling into local energy minima.

Summary by alphaXiv.