Reproduce “A Transformer with Stack Attention” on OpenResearch

About this paper

Jiaoda Li, Jennifer C. White, Mrinmaya Sachan, Ryan Cotterell

Researchers from ETH Zürich and the University of Cambridge developed a 'stack attention' mechanism for Transformer architectures to improve their handling of hierarchical language structures. This augmentation enabled near 100% accuracy on specific deterministic context-free language tasks and provided benefits for language modeling when data was limited, though it proved detrimental when fine-tuning large pre-trained models.

Key insights

• The stack-augmented Transformer successfully addresses limitations in handling certain hierarchical dependencies, achieving high accuracy on tasks like Reverse String and Stack Manipulation.
• The augmented model still struggles with formal languages requiring modular arithmetic, suggesting an inherent limitation related to 'modulo counting' capabilities that even stack augmentation cannot fully overcome.
• Stack attention provides a beneficial inductive bias for language modeling in low-data regimes but can be detrimental when fine-tuning large pre-trained models, possibly due to conflicts with their already rich learned representations.

Summary by alphaXiv.