Reproduce “TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Taesoo Kim, Jong Hwan Ko

The TESU-LLM framework, developed by researchers from KT Corporation and Sungkyunkwan University, enables Large Language Models to process speech inputs without requiring any speech data during training. It achieves this by aligning speech and text representations through a pre-trained unified encoder and demonstrates competitive performance on voice assistant benchmarks and automatic speech recognition.

Key insights

  • Speech capabilities can be achieved in Large Language Models by semantically aligning speech and text within a unified encoder, removing the need for direct speech training data.
  • Leveraging pre-trained multimodal foundation models allows for knowledge transfer across modalities without explicit multimodal supervision during LLM training.
  • Training only a small, lightweight projection layer while freezing large pre-trained components significantly reduces computational costs and data requirements for speech-capable AI.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.