Reproduce “Parallelizing Linear Transformers with the Delta Rule over Sequence Length” on OpenResearch

About this paper

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim

Researchers from MIT and MIT-IBM Watson AI Lab developed a hardware-efficient, parallelized training algorithm for DeltaNet, a linear transformer model that uses the delta rule for improved associative recall. This advancement enables DeltaNet to achieve competitive language modeling performance, surpass other linear models on specific recall tasks, and demonstrate further improvements in hybrid architectures combining it with traditional attention.

Key insights

• The delta rule is effective for enhancing associative recall and in-context learning in linear sequence models, as demonstrated by DeltaNet's performance on synthetic and real-world recall tasks.
• Combining DeltaNet layers with a small number of traditional attention layers (sliding window or global) creates hybrid architectures that achieve superior performance on recall-intensive tasks and competitive language modeling, often outperforming pure linear models and even strong Transformers.
• Hardware-aware algorithmic design, including memory-efficient reparameterization and techniques like the UT transform, is essential for making theoretically efficient recurrent neural networks practical and scalable on modern GPU hardware.

Summary by alphaXiv.