Reproduce “Fast and Exact Relative-Position Attention via Structure-Aware Kernels” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Delyan Boychev, Ivo Petrov, Dimitar I. Dimitrov

The authors develop tiled attention for relative-position attention, where query–key scores depend on positional offsets. The kernel computes only relation projections each tile uses, reuses constant-offset values, and assigns gradient sums to one writer to avoid atomic contention. In whole-talk TED-LIUM 3 transcription, Parakeet-TDT-0.6B-v3 reaches 1,103 RTFx versus NeMo SDPA’s 613, with 11 versus 27 GiB peak memory and close WER, 2.89% versus 3.01%; this result is specific to that test.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.