Reproduce “Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, Hitomi Yanaka

Researchers from The University of Tokyo, Riken, University of Liverpool, and Tohoku University developed CAROT, a method that disentangles language-specific and language-agnostic components in LLM representations and aligns the language-agnostic parts at the token level using optimal transport. This approach led to an 11.2-point accuracy improvement during inference-time steering and outperformed baseline multilingual training methods in 11 out of 18 evaluated scenarios.

Key insights

  • LLM hidden states can be effectively decomposed into distinct language-specific and language-agnostic components, and aligning only the latter is crucial for preserving linguistic fidelity.
  • Fine-grained, token-level alignment, facilitated by optimal transport, is more effective for bridging semantic gaps across languages than coarse, sentence-level interventions.
  • Interventions for cross-lingual alignment are most impactful in the early-to-middle layers of LLMs, where question tokens' information is processed before full answer generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.