Reproduce “Conformer-based End-to-end Speech Recognition With Rotary Position Embedding” on OpenResearch

About this paper

Shengqiang Li, Menglong Xu, Xiao-Lei Zhang

Researchers at Northwestern Polytechnical University's CIAIC improved Conformer-based end-to-end speech recognition by integrating Rotary Position Embedding (RoPE). This integration resulted in a Word Error Rate of 2.1% on LibriSpeech 'test-clean' and a Character Error Rate of 4.69% on AISHELL-1, outperforming baseline Conformers using other position embedding methods.

Key insights

• Rotary Position Embedding (RoPE) effectively captures and utilizes positional information by directly integrating relative position information into the self-attention mechanism via rotation matrices.
• RoPE consistently outperforms both Absolute Position Embedding (APE) and traditional Relative Position Embedding (RPE) when integrated into the Conformer architecture for ASR.
• The study validates RoPE's efficacy in a complex, state-of-the-art ASR model, suggesting its potential applicability to other Transformer-based sequence modeling tasks beyond speech.

Summary by alphaXiv.