Reproduce “The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
An analysis of transformer embedding geometry distinguishes between encoder and decoder architectures, identifying a bell-shaped anisotropy profile unique to decoders. The study also uncovers a two-phase dynamic in decoder intrinsic dimensionality during training, initially expanding representations before compressing them, which correlates with improved model performance.
Key insights
- Transformer decoders consistently exhibit a bell-shaped anisotropy profile across layers, while encoders show more uniform anisotropy, indicating fundamental architectural differences in information representation.
- Decoder intrinsic dimensionality undergoes a two-phase dynamic during training: an initial expansion into higher-dimensional space followed by a compression into more efficient representations.
- The observed intrinsic dimension dynamics in decoders are driven by deep learning processes and correlate with ongoing performance improvements, rather than being simple consequences of learning rate schedules.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.