Reproduce “Evaluating Morphological Alignment of Tokenizers in 70 Languages” on OpenResearch

About this paper

Catherine Arnett, Marisa Hudspeth, Brendan O'Connor

This research evaluates morphological alignment in tokenizers across 70 languages using an expanded MorphScore framework, developing new datasets and metrics. The findings indicate that morphological alignment, as intrinsically measured, shows a weak and often negative correlation with downstream language model performance, explaining only a negligible portion of the variance.

Key insights

• Morphological alignment, as intrinsically measured by the refined MorphScore, explains only a tiny fraction (R² between 0.005 and 0.024) of the variance in downstream language model performance.
• A consistent and robust finding is a weak, and often negative, correlation between morphological alignment and downstream model performance, challenging the common assumption that linguistically aligned tokenization inherently improves model capabilities.
• Control variables such as model size and the proportion of training data in each language remain the primary and statistically significant drivers of language model performance, far outweighing the observed impact of morphological alignment.

Summary by alphaXiv.