Reproduce “Data Scaling Laws in NMT: The Effect of Noise and Architecture” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, Orhan Firat
A large-scale study by Harvard University and Google researchers quantified the impact of architectural choices and data quality on data scaling laws in Neural Machine Translation (NMT), revealing that architecture and independent noise primarily lead to multiplicative performance shifts, whereas back-translated data notably degrades the data scaling exponent.
Key insights
- The data scaling exponent for NMT models is surprisingly robust across diverse encoder-decoder Transformer architectures and independent noise types, suggesting it captures a fundamental property of the learning problem.
- Architectural and independent noise interventions primarily cause multiplicative shifts in scaling curves, indicating that their performance benefits can often be compensated by acquiring more data rather than fundamentally changing learning efficiency.
- Dependent noise, specifically from back-translation, fundamentally alters and degrades the data scaling exponent, revealing a measurable reduction in learning efficiency compared to human-generated parallel data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.