Reproduce “BPE-Dropout: Simple and Effective Subword Regularization” on OpenResearch

About this paper

Ivan Provilkov, Dmitrii Emelianenko, Elena Voita

Researchers from Yandex, MIPT, HSE, University of Edinburgh, and University of Amsterdam developed BPE-dropout, a straightforward subword regularization method that injects stochasticity into Byte Pair Encoding segmentation during NMT training. This approach consistently yields 0.5 to 2.3 BLEU point improvements on various translation tasks and notably enhances model robustness against misspelled input by 1.6 to 2.3 BLEU points, without affecting inference time.

Key insights

• Introducing stochasticity into BPE during training allows NMT models to learn more robust and compositionally aware subword representations from diverse segmentations.
• This approach qualitatively improves token embeddings, ensuring that rare tokens are better integrated into the embedding space and exhibit stronger character-level similarity to their neighbors.
• The simplicity of BPE-dropout, which modifies the existing BPE algorithm, makes it easily adoptable without requiring new, complex segmentation methods.

Summary by alphaXiv.