Reproduce “Efficient conformer-based speech recognition with linear attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengqiang Li, Menglong Xu, Xiao-Lei Zhang
An efficient Conformer variant, the Linear Attention Based Conformer (LAC), is proposed to address computational and parameter count limitations in state-of-the-art Automatic Speech Recognition models. This architecture reduces self-attention complexity to linear time and halves the parameter count by approximately 50%, while maintaining competitive character and word error rates on AISHELL-1 and LibriSpeech datasets.
Key insights
- Adapting linear attention mechanisms from other fields, like object detection, can successfully reduce the computational complexity of self-attention in speech models to a linear scale.
- Low-rank matrix factorization is an effective technique for compressing feed-forward networks within ASR models, contributing to substantial parameter reduction with minimal performance impact.
- A good trade-off exists where state-of-the-art ASR performance can be largely maintained even with a significantly smaller model size and faster training/inference times.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.