Reproduce “Cross-Modal Knowledge Distillation for Speech Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Enzhi Wang, Qicheng Li, Zhiyuan Tang, Yuhang Jia
A cross-modal knowledge distillation framework improves speech Large Language Models (LLMs) by mitigating catastrophic forgetting and modality inequivalence. The approach elevates Qwen2.5-Omni's overall Speech-to-Text (S→T) performance to 77.19 on VoiceBench from a baseline of 75.08, while also enhancing Text-to-Text (T→T) capabilities to 79.86.
Key insights
- Catastrophic forgetting and modality inequivalence are widespread and quantifiable issues across mainstream open-source speech LLMs, significantly hindering their performance.
- A dual-channel knowledge distillation framework effectively addresses both catastrophic forgetting and modality inequivalence by leveraging a strong text LLM as a teacher.
- Using teacher-generated outputs as hard targets combined with Kullback-Leibler (KL) divergence for distillation proves effective in transferring higher-level semantic knowledge and improving reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.