Reproduce “Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaeyoung Lee, Masato Mimura
Researchers from NTT, Inc. developed a decoder-only Conformer architecture for Automatic Speech Recognition (ASR) that integrates modality-aware sparse Mixtures of Experts (MoE) and a hybrid causality scheme. This randomly initialized model achieved 2.8% Word Error Rate (WER) on LibriSpeech test-clean, outperforming larger Attention Encoder-Decoder (AED) baselines with fewer active parameters.
Key insights
- Modality-aware MoE provides effective specialization for speech and text processing within a unified decoder-only stack, outperforming generic MoE and non-MoE counterparts.
- The integration of CTC loss applied specifically to speech positions significantly enhances alignment and overall performance in decoder-only ASR.
- A randomly initialized decoder-only Conformer, when combined with modality-aware MoE and a hybrid causal design, can surpass strong AED baselines in accuracy with fewer active parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.