Reproduce “Wav2Vec-Aug: Improved self-supervised training with limited data” on OpenResearch

About this paper

Anuroop Sriram, Michael Auli, Alexei Baevski

Researchers at Facebook AI (now Meta AI) developed Wav2Vec-Aug, an enhanced version of Wav2Vec 2.0, by systematically combining data augmentation techniques with architectural improvements. This approach achieved up to a 13% relative improvement in Word Error Rate, particularly in scenarios with limited unlabeled pre-training data, addressing a key challenge for low-resource applications.

Key insights

• Data augmentation, particularly a novel dual-stream approach, can significantly benefit complex transformer-based self-supervised models like Wav2Vec 2.0, especially in low-data scenarios.
• Systematic integration of architectural advancements, such as Conformer blocks, dynamic convolutions, and projection MLPs, can yield substantial performance improvements for established self-supervised learning models.
• The research specifically addresses the challenge of data scarcity, showing that advanced self-supervised speech models can be made more data-efficient, which is critical for low-resource languages and domains.

Summary by alphaXiv.