Reproduce “IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad B, Varun Balan G, Sparsh Jain and 4 more
The IndicLLMSuite provides a comprehensive resource for 22 Indian languages, offering 251 billion tokens for pre-training and 74.7 million instruction-response pairs for fine-tuning. This initiative from AI4Bharat and IIT Madras establishes an open-source data curation pipeline to enable the development of more capable and culturally relevant Large Language Models for significantly under-resourced linguistic groups.
Key insights
- Existing multilingual corpora contain significant noise, underscoring the necessity for meticulous, language-specific data curation pipelines like SETU to effectively filter and deduplicate large volumes of data.
- A combination of human-verified, machine-translated, and synthetically generated data is crucial for achieving both high quality and sufficient scale for low-resource languages, addressing the "chicken and egg" problem.
- Instruction fine-tuning data can be effectively amassed by aggregating existing datasets, translating/transliterating English resources, and generating context-grounded conversations using open-source LLMs, alongside crowdsourcing, to impart diverse instruction-following capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.