Reproduce “Pay Less Attention with Lightweight and Dynamic Convolutions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli

Researchers from Cornell University and Facebook AI Research introduced lightweight and dynamic convolutions, achieving a new state-of-the-art BLEU score of 29.7 on WMT'14 English-German machine translation and demonstrating up to 20% faster inference compared to self-attention models across several sequence modeling tasks. This work shows that efficient, convolution-based architectures can match or exceed the performance of content-based self-attention.

Key insights

  • Sophisticated content-based self-attention is not strictly necessary for achieving state-of-the-art results in sequence modeling, as demonstrated by the strong performance of simplified convolutional architectures.
  • Dynamic kernel generation, where kernels are adapted per time step based on local input, offers an efficient way to capture context without the quadratic computational cost of global self-attention.
  • Softmax-normalization of convolution kernel weights is critical for the stability and effective training of dynamically generated kernels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.