Reproduce “Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rachel S.Y. Teo, Tan M. Nguyen
A theoretical framework unveils self-attention as projections onto principal components within a feature space via Kernel Principal Component Analysis. This KPCA-based understanding informs "Attention with Robust Principal Components (RPC-Attention)", which enhances transformer resilience against common corruptions and adversarial attacks, demonstrating improved accuracy across ImageNet-C, ImageNet-A, ImageNet-O datasets, and various adversarial benchmarks.
Key insights
- Self-attention fundamentally operates as a Kernel Principal Component Analysis, projecting query vectors onto the principal components of key vectors in an induced feature space.
- The value matrix in self-attention is not merely a linear transformation but represents the eigenvectors of the Gram matrix of key vectors in the feature space.
- Integrating robust PCA techniques systematically enhances the resilience of self-attention layers against various data corruptions and adversarial perturbations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.