Reproduce “Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhen Ye, Xu Tan, Yiming Li, Guangyan Zhang, Chimin Chan, Haohe Liu, Zhengxi Liu, Hongzhan Lin and 5 more
A study systematically investigated how speech token frame rate and alignment depth affect reasoning transfer in large language models designed for text, introducing a factorized quantization method. It identified an optimal frame rate range (4.17-6.25 Hz) for improved speech question answering performance, showing that middle-layer alignment is most effective.
Key insights
- An optimal speech token frame rate exists (approximately 4.17-6.25 Hz), slightly above the average text token rate, for maximizing cross-modal reasoning transfer in LLMs.
- Factorized FSQ effectively overcomes information bottlenecks at low frame rates, maintaining robust ASR performance and high reconstruction quality without requiring explicit codebooks.
- Contrastive alignment is most effective when applied at intermediate LLM layers (specifically the middle layer, L/2) to bridge the semantic modality gap, outperforming alignment at input embedding or other layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.