Reproduce “Better speech synthesis through scaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
James Betker
TorToise, developed by an independent researcher, applies techniques from large-scale image generation to achieve highly realistic and expressive multi-voice text-to-speech synthesis. It demonstrates that combining autoregressive models for text-to-token mapping with diffusion models for high-fidelity spectrogram generation, alongside a massive 49,000-hour dataset, significantly advances speech quality.
Key insights
- Methodologies successful in large-scale image generation (generalist architectures, massive datasets, large-scale training) are effectively transferable and potent for speech synthesis.
- Combining autoregressive models (for unaligned domain mapping like text-to-speech tokens) with DDPMs (for generating high-quality continuous data like spectrograms) is a robust strategy for complex generative tasks.
- High-quality, expressive speech synthesis can be achieved by leveraging open-source tools and publicly available datasets, even by independent researchers with limited computational resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.