Reproduce “Noisy Training Improves E2E ASR for the Edge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dilin Wang, Yuan Shangguan, Haichuan Yang, Pierce Chuang, Jiatong Zhou, Meng Li, Ganesh Venkatesh, Ozlem Kalinli and 1 more
Facebook AI introduces "noisy training," a method that injects Gaussian noise into model parameters during training, to improve the generalization of End-to-End ASR models, particularly when compressed for edge device deployment. The approach consistently reduced Word Error Rate, with up to 14% relative reduction on highly pruned models, by encouraging smoother parameter landscapes.
Key insights
- Injecting Gaussian noise directly into model parameters encourages the learning algorithm to find smoother, more generalizable solutions in the parameter space.
- Noisy training leads to lower validation loss and produces more uniform singular value distributions in learned weight matrices, suggesting better utilization of higher-dimensional representation spaces.
- The technique is complementary to existing regularization methods like dropout and data augmentation, addressing different aspects of overfitting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.