Reproduce “Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis” on OpenResearch

About this paper

Aman Kassahun Wassie, Mahdi Molaei, Yasmin Moslem

This study empirically compares open-source large language models against a specialized machine translation model (NLLB-200 3.3B) for medical domain translation across high and low-resource language pairs. It finds that fine-tuned NLLB-200 often outperforms smaller LLMs and competes with massive LLMs, particularly for lower-resource languages, highlighting the continued relevance of specialized MT systems for high-quality domain-specific tasks.

Key insights

• Specialized encoder-decoder MT models like NLLB-200 remain essential for high-quality, domain-specific translation, often outperforming general LLMs when fine-tuned on relevant data.
• Retrieval-augmented one-shot prompting consistently enhances translation quality for autoregressive decoder-only LLMs by leveraging in-context learning.
• Large Language Models continue to struggle significantly with domain-specific translation in lower-resource languages, demonstrating a clear performance gap compared to specialized MT models.

Summary by alphaXiv.