Reproduce “Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi, Jarod Alper
This research reveals how small transformers compute modular integer multiplication over composite moduli, an operation characterized by non-invertible elements, by identifying a 'Monoid Extension' where inputs are partitioned into algebraic J-classes. The models then apply localized Fourier-based mechanisms within these strata, explaining up to 99.4% of output logit variance using local group characters.
Key insights
- Transformers learn to stratify algebraic structures, partitioning the input space into J-classes and processing them using localized, group-like computations.
- The embedding layer encodes input elements into low-dimensional, J-class-specific subspaces with sparse Fourier features, demonstrating an 'atomic factorization hypothesis'.
- Attention mechanisms perform algebraic routing, with some heads (e.g., Head 0's QK circuit) acting as macroscopic routers for J-classes, and OV circuits functioning as low-rank projection filters for key Fourier frequencies.
- The model's output logits are well-explained by local group characters, utilizing local inverses and identities within each J-class for a character-based readout.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.