Reproduce “Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding” on OpenResearch

About this paper

Seungwoo Choi, Seungju Han, Dongyoung Kim, Sungjoo Ha

Attentron, a few-shot text-to-speech system from Hyperconnect, clones the voices of unseen speakers using a dual-encoder architecture and attention-based variable-length embeddings. It enhances speaker similarity and synthesis stability by effectively integrating multiple reference audio samples.

Key insights

• A dual-encoder architecture effectively addresses both synthesis stability (coarse-grained encoder) and detailed speaker style capture (fine-grained encoder) for few-shot unseen speaker cloning.
• Leveraging multiple reference audio samples through an attention mechanism significantly improves speaker similarity and synthesis quality, with performance scaling with more available references.
• Minimally processing the reference audio to derive attention values (from raw spectrograms via a single fully-connected layer) encourages generalization and imitation of new voices, rather than memorization of training speakers.

Summary by alphaXiv.