Reproduce “Integral Transformer: Denoising Attention, Not Too Much Not Too Little” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ivan Kobyzev, Abbas Ghaddar, Dingtao Hu, Boxing Chen
The Integral Transformer introduces a new self-attention mechanism that integrates signals from logit distributions to effectively denoise attention. This approach reduces excessive focus on uninformative tokens while preserving essential contextual information, leading to consistent performance improvements across various language understanding and reasoning benchmarks.
Key insights
- Attention denoising is most effective when applied to the upper layers of a Transformer, indicating that attention noise impacts more abstract or global representations processed in deeper layers.
- A balanced denoising approach that reduces but does not fully eliminate attention to special tokens yields superior performance, supporting the idea that these tokens retain crucial contextual roles.
- Integrating logit signals through averaging provides a robust method to mitigate attention noise, leading to more balanced attention distributions and improved effective rank in attention matrices.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.