Reproduce “Efficient Domain Adaptation for Speech Foundation Models” on OpenResearch

About this paper

Bo Li, Dongseong Hwang, Zhouyuan Huo, Junwen Bai, Guru Prakash, Tara N. Sainath, Khe Chai Sim, Yu Zhang and 3 more

Researchers at Google LLC developed an efficient recipe for adapting speech Foundation Models to new domains, achieving a 4.4% Word Error Rate on Voice Search, comparable to a baseline model that required 15 times more supervised in-domain data and 5 times more finetuned parameters. The approach integrates self-supervised pretraining, unsupervised target domain audio data, and parameter-efficient encoder adapters to deliver high-quality Automatic Speech Recognition with significantly reduced resource demands.

Key insights

• Freezing the bulk of a large speech FM and adapting only small residual encoder adapters and the decoder can achieve near state-of-the-art ASR quality while updating only a small fraction (17.9%) of the model's parameters.
• Unsupervised target domain audio data, effectively integrated through methods like JUST Hydra, significantly improves domain adaptation by allowing the model to adapt to acoustic characteristics without relying on transcriptions.
• Employing a strong teacher model in Noisy Student Training can leverage vast amounts of unsupervised data for pseudo-labeling, yielding high accuracy even when using a limited subset of that unsupervised data.

Summary by alphaXiv.