Reproduce “Dynamic Short Convolutions Improve Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Oliver Sieberling, Bharat Runwal, Rameswar Panda, Yoon Kim

The authors introduce dynamic short convolutions that predict an input-dependent filter for each token and apply it over a causal window, adding the operation to Transformer queries, keys, and values. On associative recall, the method outperforms static-convolution and vanilla Transformers. At 2B scale, it scores 59.70 versus 58.97 for a parameter-matched static baseline; scaling-law compute advantages are fitted through 2B models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.