Reproduce “Attention-based clustering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rodrigo Maulen-Soto, Pierre Marion, Claire Boyer
This work theoretically analyzes simplified Transformer attention layers, demonstrating their ability to perform unsupervised clustering by recovering Gaussian Mixture Model centroids. It shows that attention acts as an approximate quantizer and possesses an intrinsic capacity for dynamic, in-context clustering even without trainable parameters.
Key insights
- Attention head parameters can be optimized via Projected Gradient Descent to align with and recover the true underlying GMM centroids in a fully unsupervised manner.
- Attention layers function as approximate quantizers, capable of denoising input observations and reducing variance, and can even achieve a lower asymptotic risk than an optimal quantizer for infinite sequence lengths.
- The core attention mechanism possesses an intrinsic capacity for dynamic, in-context approximate quantization, adapting to sequence-specific distributions without any trainable parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.