Reproduce “Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhengxuan Wu, Alex Tamkin, Isabel Papadimitriou

The paper "Oolong" investigates the specific linguistic factors that make cross-lingual transfer challenging for pretrained language models. Researchers from Stanford University found that aligning vocabulary and managing the embedding matrix is the most difficult aspect, significantly more so than adapting to syntactic changes or tokenizer quality, particularly in low-data settings.

Key insights

  • Pretrained language models demonstrate remarkable flexibility in adapting to significant syntactic word-order changes, even with limited target-language data.
  • The primary bottleneck in low-data cross-lingual transfer is the misalignment or re-initialization of the layer 0 word embedding matrix.
  • Tokenizer quality has a minimal additional impact on transfer difficulty when the foundational embedding alignment is already compromised, making it a "lower-order bit."

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.