Reproduce “Conditions for Catastrophic Forgetting in Multilingual Translation” on OpenResearch

About this paper

Danni Liu, Jan Niehues

A systematic empirical study identifies the conditions under which catastrophic forgetting occurs in multilingual machine translation, revealing that the relative scale between model and fine-tuning data, instruction-following abilities, and language control mechanisms are key factors. The work demonstrates that cross-lingual alignment and in-language instructions effectively mitigate forgetting, while Parameter-Efficient Fine-Tuning (LoRA) did not offer a clear advantage.

Key insights

• The relative scale between model size and fine-tuning data volume is a critical determinant of forgetting severity; larger models show greater resistance, but higher data volume intensifies forgetting.
• Instruction-following ability in models provides stronger protection against target language forgetting than specific architectural choices, as demonstrated by an ablation study on Qwen2.5-Instruct.
• LoRA, applied to all self-attention and linear projection components, resulted in comparable levels of catastrophic forgetting to full fine-tuning under the experimental conditions.

Summary by alphaXiv.