Reproduce “BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani
The BHARATI project developed a suite of morphology-aware subword tokenizers for seven Indian languages and English, demonstrating improved tokenization efficiency for classical Indian languages by reducing token counts by up to 90% compared to general-purpose tokenizers. This optimization significantly enhances the effective context length and training efficiency of small language models, particularly for educational AI applications in Indian Knowledge Systems.
Key insights
- Integrating morphology-aware preprocessing, such as sandhi resolution for Sanskrit and suffix boundary hints for Tamil, enables the BPE algorithm to generate more linguistically meaningful and efficient subword segmentations.
- A balanced corpus composition during tokenizer training is critical to prevent high-resource languages (e.g., English) from dominating vocabulary allocation, ensuring equitable native subword coverage for diverse Indic scripts.
- Improvements in tokenizer fertility and vocabulary utilization directly correlate with significant reductions in language model training loss and increased effective character coverage, validating the tokenizer's downstream impact.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.