Reproduce “Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, Gábor Bella
The authors introduce umLabeller, which classifies tokenizer splits using morphological data, and an Out-of-Vocabulary Generalization Challenge testing unfamiliar words in three classification tasks. Across models, examples labeled morph receive higher accuracy than alien examples. On WaD, DeBERTa-large scores 75.5 ± 2.8% on morph examples and 68.4 ± 2.5% on alien examples; these subgroup comparisons do not isolate tokenizer effects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.