Reproduce “Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyunsik Kim, Youngmoon Jung
The authors present UBE, which routes Unicode characters by UTF-8 length while retaining standard byte-level BPE: one- and two-byte characters remain UTF-8, while three- and four-byte characters use UTF-16LE. This lowers the encoding floor for three-byte BMP characters from three symbols to two. On FLORES-200, for 101 languages and a 32K vocabulary, token-premium Gini falls 9.4%; these tokenizer results do not establish better model quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.