Reproduce “Joint ASR and Speaker Role Tagging with Serialized Output Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anfeng Xu, Tiantian Feng, Shrikanth Narayanan
This work introduces a unified approach for joint Automatic Speech Recognition and Speaker Role Tagging by fine-tuning the Whisper foundation model using Serialized Output Training. The proposed method achieved a 10-15% reduction in multi-talker Word Error Rate compared to a strong WavLM-large baseline across two conversational datasets, demonstrating robust capabilities in transcribing speech and identifying speaker roles such as child/adult or self/other.
Key insights
- Fine-tuning Whisper's encoder layers is essential for adapting acoustic representations to speaker-specific characteristics and domain environments, as freezing them led to substantial performance degradation.
- A maximum audio input duration of 15 seconds provides sufficient context for effective joint ASR and Speaker Role Tagging, with longer inputs yielding minimal additional performance benefits.
- Conditioning on previously predicted tokens unexpectedly degraded performance, likely due to a train-inference mismatch where models trained on ground-truth context must rely on potentially erroneous predictions during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.