Reproduce “StableMask: Refining Causal Masking in Decoder-only Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, Qiang Zhang
StableMask introduces a parameter-free method that refines causal masking in decoder-only Transformers, addressing issues of disproportional attention distribution and the inability of Relative Positional Encoding (RPE) to capture absolute positional information. The approach, which involves introducing pseudo-attention scores with positional decay, improved perplexity on datasets like MiniPile (e.g., OpenLLaMA-SM RoPE 160M achieved 25.0 PPL compared to 25.9 for the baseline), enhanced extrapolation capabilities, and boosted performance on downstream tasks.
Key insights
- Modifying the softmax output by introducing pseudo-attention scores causes the sum of real attention scores per row to deviate from one, thereby enabling the model to explicitly encode absolute positional information without adding parameters.
- The position-adaptive decay characteristic of pseudo-attention scores effectively serves as a dynamic 'attention sink,' absorbing excess attention without relying on fixed artificial tokens, leading to more stable attention distributions.
- The parameter-free nature of StableMask ensures that it can be seamlessly integrated into existing RPE-based Transformer architectures, preserving their extrapolation benefits while concurrently addressing attention distribution and absolute position encoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.