Reproduce “Closing the Gap Between Text and Speech Understanding in LLMs” on OpenResearch

About this paper

Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

A two-stage training methodology, SALAD (Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation), enables speech-adapted Large Language Models (LLMs) to close the text–speech understanding gap. It achieves competitive performance on broad-domain knowledge and reasoning tasks while demonstrating superior data efficiency, utilizing over an order of magnitude less public speech data than comparable models.

Key insights

• Cross-modal misalignment and catastrophic forgetting are quantifiable factors that strongly predict the text–speech understanding gap in speech-adapted LLMs.
• Cross-modal knowledge distillation significantly outperforms standard negative log-likelihood (NLL) for mitigating both misalignment and forgetting during speech adaptation.
• Actively selecting synthetic speech data based on measured domain-specific misalignment is a more efficient strategy than uniform sampling for improving generalizability and closing performance gaps.

Summary by alphaXiv.