Reproduce “Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward” on OpenResearch

About this paper

Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioğlu, Pradeep Rangappa, Petr Motlicek and 2 more

An empirical evaluation of the SLAM-ASR architecture demonstrates its efficiency and in-domain performance, while also identifying its significant vulnerability to domain shifts and speech perturbations. Integrating LoRA for LLM fine-tuning dramatically improves its cross-domain generalization and speech-to-text alignment.

Key insights

• SLAM-ASR's constrained training, involving only the projector layer, leads to overfitting to its training domain, resulting in poor generalization and severe performance degradation on out-of-domain data.
• The architecture demonstrates high sensitivity to acoustic perturbations, such as altered speaking rates (especially slower rates) and background noise, indicating limited capacity to handle real-world acoustic variability.
• Fine-tuning the LLM with LoRA adapters is crucial for learning effective and monotonically aligned speech-to-text representations, which substantially improves both in-domain and cross-domain robustness.

Summary by alphaXiv.