Reproduce “ConvPE: Convolutional Position Embedding Improves Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zehao Wang
The authors introduce ConvPE, which applies a content-dependent local bias to attention scores using projected token states and a causal convolution; keys beyond its window receive no bias from this mechanism. In a single-run comparison, ConvPE reaches the RoPE baseline’s loss after 2.06 billion tokens rather than 3 billion. Models contain up to 167 million parameters, limiting claims about reproducibility and scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.