Reproduce “Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Taimoor Hassan, Jawad Ahmed, Muhammad Awais
Qalb introduces a new state-of-the-art Urdu Large Language Model for over 230 million speakers, developed through systematic continued pre-training on a 1.97 billion token Urdu corpus. It achieved an overall weighted average score of 90.34 across diverse NLP tasks, outperforming the previous state-of-the-art model by 3.24 points and its base model by 44.64 points.
Key insights
- Systematic and extensive continued pre-training on a substantial, high-quality target-language dataset is crucial for adapting powerful foundation models to low-resource languages, yielding dramatic performance improvements.
- A mixed dataset strategy, incorporating some original-language data during continued pre-training, effectively prevents catastrophic forgetting and helps retain the base model's general reasoning abilities.
- Utilizing LLMs as 'automatic judges' (e.g., GPT-4o) for evaluation proves to be a reliable and scalable method for assessing model outputs, with human validation confirming over 85% agreement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.