Reproduce “MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, Hinrich Schütze
The MURI method generates high-quality instruction tuning datasets for 200 languages, particularly low-resource ones, by leveraging human-written texts as outputs and reverse-generating instructions. Models fine-tuned on the resulting MURI-IT dataset demonstrate improved multilingual Natural Language Understanding and Natural Language Generation capabilities compared to previous baselines.
Key insights
- Sourcing authentic human-written text for outputs avoids "translationese" artifacts and preserves linguistic and cultural nuances in instruction tuning datasets.
- Reverse instruction generation, using a high-resource language LLM to create instructions from translated outputs, effectively generates diverse instructions for various topics.
- Combining the MURI method with curated existing NLP tasks and quality-controlled web data creates a comprehensive and diverse instruction tuning dataset for a wide range of languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.