Reproduce “Generative Expressive Conversational Speech Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, Haizhou Li
GPT-Talker synthesizes supplied response text with speech style suited to preceding conversation, rather than choosing what to say. It serializes dialogue text and audio as alternating tokens, uses GPT to predict response acoustic tokens, then combines them with the given words and a speaker reference. On English tests, listener ratings exceed ECSS on DailyTalk; ratings do not establish natural speech.
Key insights
- Representing dialogue history as one sequence of text and audio tokens offers a way to model conversational meaning and speaking style without a separate collection of context-specific modules.
- Turn-by-turn alternation of text and audio performs better than grouping all text apart from all audio, and the tested three-turn context works better than shorter or longer contexts.
- Staged training generally outperforms training only on NCSSD, while voice references enable synthesis for speakers absent from training, though similarity is lower for those speakers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.