Reproduce “Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen

The authors introduce a spoken-language model that connects a text language model to a speech encoder, aiming to answer spoken input and written instructions directly. During training, it matches speech-query length to transcript length and aligns corresponding representations while retaining response-based training. It reaches CKA 0.6399 and SpeechR accuracy 52.91, but higher alignment does not always improve task performance.

Key insights

  • Existing SLMs, despite strong task performance, produce speech representations that remain structurally misaligned with text embeddings.
  • Decoupling the length matching problem from semantic alignment via dynamic query allocation significantly enhances the modality adapter's ability to learn semantic mapping.
  • Token-level internal alignment loss, when optimally weighted, fosters deeper representational similarity between speech and text, leading to improved SLM instruction-following and generalization.
  • The model retains the ability to capture and utilize speech-specific cues (e.g., emotions) even with token-level internal alignment to text.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.