Reproduce “SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyun Fan, Linhao Dong, Jun Zhang, Lu Lu, Zejun Ma
SA-SOT from Bytedance Research addresses the "semantic confusion problem" in token-level Serialized Output Training (t-SOT) for multi-talker ASR by integrating speaker-aware mechanisms into the decoder, achieving a new state-of-the-art concatenated minimum-permutation word error rate of 3.41% on LibrispeechMix.
Key insights
- The semantic confusion in Serialized Output Training (SOT) can be resolved by making the autoregressive decoder explicitly speaker-aware.
- Speaker identity information can be effectively integrated into the ASR decoder through direct embedding fusion, auxiliary masked training, and speaker-aware attention mechanisms.
- Modifying self-attention based on speaker similarity allows the model to emphasize same-speaker context and diminish cross-speaker interference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.