Reproduce “StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaicheng Luo, Xuefei Gong, Yutao Sun, Jinling He, Yujie Hou, Xiaoyang Xing, Huiyan Li, Bing Han and 1 more
StellarTTS introduces a non-autoregressive framework that generates robust, low-latency, and prosodically natural speech by integrating a sparse temporal embedding, a semantic-aware codec, and a mobile-optimized LLaMA-based decoder. The system achieves a Real-Time Factor of 0.08 and a Word Error Rate of 7.47 on challenging datasets, outperforming prior methods in speed and linguistic accuracy.
Key insights
- Anchoring linguistic information at sparse temporal points within phone durations can enhance both prosodic naturalness and linguistic robustness without requiring strict frame-wise alignments.
- Unifying semantic and acoustic representation learning within a single codec and decoding stage significantly reduces pipeline complexity and inference latency, beneficial for resource-constrained environments.
- Designing the entire TTS architecture with a mobile-first approach, including a lightweight LLaMA-based decoder and optimized decoding steps, enables superior real-time performance on edge devices.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.