Reproduce “The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jacob Brinton, Jannik Brinkmann, Mark Crovella, Aaron Mueller
Evidence from Llama-3.1-8B and Aya-23-8B supports an interlingua-like mechanism in which translation reuses language-general representations and components that also serve monolingual tasks. Monolingual competence explains much of the variation in translation quality, shared attention heads affect many language pairs, and monolingual Xhosa adaptation recovers up to 99% of the gain from parallel-data fine-tuning into English.
Key insights
- Source and target language competence largely explains pairwise translation differences: additive language effects account for 93.2% of centered BLEU variance in Llama and 87.9% in Aya, while source-target interactions add little.
- Translation relies on attention heads reused across languages and located in mid-to-late layers; removing the ten strongest heads harms BLEU about twice as much as matched random ablations and also affects general task performance.
- The findings favor an interlingua-like mechanism as an important component of LLM translation, but shared causal heads and cross-lingual transfer do not establish a single universal representation or eliminate translation-specific processing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.