Reproduce “Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyi Xu, Kaixun Huang, Pengcheng Guo, Yu Zhou, Longtao Huang, Hui Xue, Lei Xie
Researchers from Northwestern Polytechnical University and Alibaba Group developed a rehearsal-free method for continually adapting large multilingual Automatic Speech Recognition (ASR) models like Whisper to new languages without degrading performance on previously learned ones. Their approach, based on Low-Rank Adaptation (LoRA) and orthogonal gradient descent (O-LoRA, O-AdaLoRA), achieved a significant reduction in catastrophic forgetting on Chinese (e.g., Chinese WER of 11.51%) while maintaining competitive performance on newly added languages (Uyghur and Tibetan), all with a highly parameter-efficient setup.
Key insights
- Orthogonal gradient descent, when applied to LoRA parameters, effectively mitigates catastrophic forgetting in multilingual ASR without requiring rehearsal data.
- LoRA parameters can capture the gradient subspace of past tasks, enabling subsequent learning to be guided orthogonally to preserve prior knowledge.
- AdaLoRA's dynamic rank allocation allows for even greater parameter efficiency in continual learning, achieving strong performance while using fewer trainable parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.