Reproduce “Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba” on OpenResearch

About this paper

Baher Mohammad, Magauiya Zhussip, Stamatios Lefkimmiatis

The MAVE architecture, developed by researchers at MTS AI and ITMO University, integrates structured state-space models (Mamba) with cross-attention for high-fidelity speech editing and zero-shot text-to-speech synthesis. This model achieves human-parity naturalness in edited speech, outperforming prior autoregressive and diffusion models in perceptual quality while demonstrating significantly reduced memory consumption.

Key insights

• Mamba-based state-space models can be successfully applied to text-conditional speech generation tasks, offering efficient, linear-time complexity for processing long audio sequences.
• Explicit cross-attention is crucial for precise linguistic conditioning within a Mamba-based audio decoder, significantly outperforming approaches that simply concatenate text and audio tokens.
• A causal masking and token rearrangement strategy enables autoregressive models to effectively achieve context-aware, bidirectional processing for robust and high-quality speech editing.

Summary by alphaXiv.