Reproduce “Long-Context Speech Synthesis with Context-Aware Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhipeng Li, Xiaofen Xing, Jingyuan Xing, Hangrui Hu, Heng Lu, Xiangmin Xu
Researchers from South China University of Technology and Alibaba Group developed a Context-Aware Memory (CAM)-based long-context Text-to-Speech model that significantly improves prosodic expressiveness, consistency, and naturalness in paragraph-level speech. The method achieved a Mean Opinion Score (MOS) of 3.796 and a Consistency MOS (CoMOS) of 3.992, outperforming state-of-the-art methods while maintaining efficient context processing with a fixed prefix length.
Key insights
- The dynamic updating and retrieval mechanism of the CAM block effectively balances long-term contextual consistency with local contextual nuances, making context utilization more efficient and adaptive.
- Speech context (Mem-S) was found to contribute more significantly to overall coherence and timbre consistency than text context (Mem-T), highlighting the importance of rich acoustic information for long-form synthesis.
- Using a Prefix Mask on contextual input tokens in an LLM, instead of a strict causal mask, enhances the model's ability to fully leverage contextual information for improved in-context learning and more accurate prosodic and stylistic predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.