Reproduce “Learning Language Specific Sub-network for Multilingual Machine Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zehui Lin, Liwei Wu, Mingxuan Wang, Lei Li
The LaSS (Language Specific Sub-network) method, developed by ByteDance AI Lab, learns and utilizes dynamic language-specific subnetworks within a single multilingual Transformer model. This approach improves translation quality for various language pairs, notably for high-resource ones and in zero-shot scenarios, by mitigating parameter interference without increasing total model parameters.
Key insights
- Language-specific sub-networks can be effectively learned and utilized within a single multilingual NMT model without adding new parameters, leading to improved translation performance and parameter efficiency.
- The method significantly alleviates the "off-target" issue in zero-shot translation, with the decoder's language-specific parameters providing a strong signal for correct target language generation.
- Language-specific capacity is concentrated in the bottom and top layers of both the encoder and decoder, and mask similarity correlates positively with language family similarity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.