Reproduce “Alternative positional encoding functions for neural transformers” on OpenResearch

About this paper

Ezequiel Lopez-Rubio, Macoris Decena-Gimenez, Rafael Marcos Luque-Baena

Researchers from the University of Málaga evaluated non-sinusoidal periodic functions for positional encoding in Transformer models, finding that triangular and sawtooth wave functions substantially improve performance. These alternatives achieved over 11 BLEU points higher on the Multi30K English-German machine translation benchmark compared to the standard sinusoidal approach.

Key insights

• The long-standing assumption that sinusoidal functions are the optimal choice for fixed positional encoding in Transformers is empirically challenged, demonstrating that other periodic functions can yield superior results.
• Alternative functions with properties like uniform output distribution (triangular, sawtooth) or quantized states (square) can interact more effectively with the self-attention mechanism, leading to improved representation learning.
• Certain alternative functions, such as the triangular wave, not only improve performance but also facilitate faster learning and convergence, suggesting potential for energy savings and reduced training times.

Summary by alphaXiv.