Reproduce “Efficient Speech Enhancement via Embeddings from Pre-trained Generative Audioencoders” on OpenResearch

About this paper

Xingwei Sun, Heinrich Dinkel, Yadong Niu, Linzhang Wang, Junbo Zhang, Jian Luan

Xiaomi's MiLM Plus researchers develop an efficient speech enhancement framework that freezes pre-trained audioencoders and vocoders while training only a compact denoise encoder (14.2M parameters) to operate on embeddings, achieving 3.87 MOS compared to 3.11 for Demucs baseline in subjective evaluations, with their key finding that generative audioencoders like Dasheng preserve superior speaker fidelity (0.783-0.881 cosine similarity) versus discriminative models like WavLM and Whisper (0.406-0.489) which discard speaker-specific characteristics despite maintaining phonetic quality, demonstrating that a Vision Transformer processing embeddings between frozen components can deliver state-of-the-art perceptual quality while enabling modular reuse across audio processing tasks and reducing computational costs compared to end-to-end fine-tuning approaches.

Key insights

• Generative audioencoders significantly outperform discriminative ones for speech enhancement, particularly in preserving speaker fidelity and holistic speech characteristics
• Discriminative models optimize for phonetic/linguistic features but inadvertently discard crucial speaker-specific information like timbre and pitch
• Substantial parameter reduction in the denoise encoder (from 14.2M to 1.2M parameters) maintains good perceptual quality despite lower intrusive metrics
• Operating on embedding space rather than raw audio enables efficient modular design with frozen pre-trained components

Summary by alphaXiv.