Reproduce “Speak While You Think: Streaming Speech Synthesis During Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Avihu Dekel, Slava Shechtman, Raul Fernandez, David Haws, Zvi Kons, Ron Hoory
IBM Research developed LLM2Speech, a system that enables high-fidelity, low-latency speech synthesis directly from Large Language Model (LLM) text generation. It achieves speech quality comparable to non-streaming systems while providing timely spoken output for fluent conversational AI interactions.
Key insights
- Achieving streaming speech synthesis quality comparable to non-streaming, full-context systems is possible through a knowledge distillation approach for PnP prediction, even with limited lookahead.
- LLM hidden embeddings provide valuable contextual information that significantly contributes to Grapheme-to-Phoneme (G2P) accuracy in a streaming environment, helping to compensate for limited future text context.
- Finetuning the PnP adaptor on a conversational speech corpus is crucial for enhancing overall naturalness and handling expressive elements like interjections in the synthesized speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.