Reproduce “Kuramoto Attention: Synchronizing Self-Attention on the Torus” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joshua Nunley
Indiana University Bloomington researchers developed Kuramoto Attention, a self-attention mechanism that represents token states as phase oscillators and updates them through Kuramoto synchronization. The model demonstrates competitive performance in byte-level language modeling, achieving lower bits per character (BPC) than a standard Transformer on CodeParrot (mean test BPC of 1.127 vs. 1.137 at 5M parameters) and comparable results on enwiki8, while offering a dynamically interpretable framework for attention.
Key insights
- Explicitly formulating self-attention as adaptive synchronization offers a framework for analyzing language model computations using concepts from dynamical systems theory.
- The learned metric gates for selection and the circular-mean tangent update are critical for the model's performance, highlighting the importance of phase alignment weighting and dynamic phase updates.
- Kuramoto Attention models can exhibit robust local synchronization among attended tokens while maintaining overall phase diversity and show depth-dependent phase-drift rates reflecting varied context integration timescales.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.