Reproduce “Languages are Modalities: Cross-Lingual Alignment via Encoder Injection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rajan Agarwal, Aarush Gupta

LLINK (Latent Language Injection for Non-English Knowledge) enhances large language model (LLM) performance for low-resource, non-Latin script languages by treating them as auxiliary modalities rather than directly tokenized input. The method achieved a 4.3x improvement in Recall@1 for Khmer-to-English retrieval and reduced decoder token count for Khmer by approximately 3x, while improving semantic understanding.

Key insights

  • Treating low-resource languages as auxiliary modalities, analogous to vision or speech inputs, enables efficient semantic injection into LLMs without the need for tokenizer modification or full model retraining.
  • Bypassing the tokenizer fragmentation problem through direct alignment of foreign embeddings to the decoder's latent space is critical for drastically improving cross-lingual retrieval and semantic understanding.
  • An explicit usage-enforcement objective is necessary during fine-tuning to ensure the LLM's decoder effectively processes and relies on the injected foreign language signal rather than disregarding it.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.