Reproduce “Efficient Scaling for LLM-based ASR” on OpenResearch

About this paper

Bingshen Mu, Yiwen Shao, Kun Wei, Dong Yu, Lei Xie

Tencent AI Lab developed the EFIN (Encoder First INtegration) training strategy for LLM-based Automatic Speech Recognition, which prioritizes independent speech encoder optimization before large language model integration. This approach reduces computational costs by 49.9% while achieving a 21.1% relative character error rate reduction compared to strong baselines, and also establishes neural scaling laws for LLM-ASR performance prediction.

Key insights

• Independently optimizing the speech encoder prior to LLM integration significantly improves both scaling efficiency and overall LLM-ASR performance.
• Only preliminary convergence of the projection layer during the alignment stage is sufficient and more computationally efficient than pushing for full convergence.
• Applying a full joint training stage that optimizes the encoder, projection, and LLM after initial encoder fine-tuning yields negligible performance gains while incurring substantial computational costs.

Summary by alphaXiv.