Reproduce “Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou

Unicoder improves multilingual transfer by pre-training one encoder with multiple cross-language objectives and fine-tuning it on source data plus machine-translated examples in other languages. The objectives target word alignment, sentence meaning, and document context. On XNLI, it averages 78.5% versus 77.8% for XLM under matched fine-tuning, with small ablation drops; translation remains necessary.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.