Reproduce “Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, Gábor Bella

The authors introduce umLabeller, which classifies tokenizer splits using morphological data, and an Out-of-Vocabulary Generalization Challenge testing unfamiliar words in three classification tasks. Across models, examples labeled morph receive higher accuracy than alien examples. On WaD, DeBERTa-large scores 75.5 ± 2.8% on morph examples and 68.4 ± 2.5% on alien examples; these subgroup comparisons do not isolate tokenizer effects.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.