Reproduce “Multi-view Subword Regularization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyi Wang, Sebastian Ruder, Graham Neubig
A new method, Multi-view Subword Regularization (MVR), enhances multilingual pre-trained language models by making them more robust to sub-optimal subword segmentation during fine-tuning. MVR achieves consistent performance gains of up to 2.5 points on the XTREME benchmark across various tasks and models, particularly benefiting non-Latin script and over-segmented languages.
Key insights
- MVR effectively mitigates the pretraining-fine-tuning discrepancy observed with simpler subword regularization, improving performance on the fine-tuning language (English).
- The consistency loss component acts as a form of label smoothing, preventing overconfident predictions and improving model calibration.
- MVR provides greater gains for languages and words that are more "over-segmented" or utilize non-Latin scripts, demonstrating its targeted effectiveness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.