Reproduce “Tokenization: A Survey for Modern NLP” on OpenResearch

About this paper

Marco Cognetta, Christopher Akiki, Pawan Sasanka Ammanamanchi, Catherine Arnett, Thomas Bauwens, Pavel Chizhov, Pieter Delobelle, Konstantin Dobler and 24 more

Tokenization is presented as a core language-model design choice that shapes sequence length, computational cost, multilingual equity, evaluation, and security—not merely as preprocessing. The survey reviews established subword methods and emerging byte-level, latent, and visual alternatives, finding that BPE still dominates practice while no tokenizer consistently optimizes compression, linguistic alignment, downstream quality, and crosslingual fairness at once.

Key insights

• BPE remains the dominant choice for generative language models, largely because of efficient implementations, ecosystem support, compatibility, and the high cost of changing tokenizers after model training—not because it is universally optimal.
• Tokenization creates measurable multilingual and economic disparities: languages with less favorable segmentation can require longer sequences, increasing attention cost, latency, and usage-based inference expense.
• Tokenizer assessment must be multidimensional. Compression, fertility, bits-per-byte, morphology, crosslingual parity, and downstream accuracy capture different properties, and intrinsic metrics do not reliably predict task performance on their own.

Summary by alphaXiv.