Reproduce “From Language Models over Tokens to Language Models over Characters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, Ryan Cotterell
Researchers at ETH Zürich, Mila, CUNY, and McGill University developed principled algorithms to transform token-level large language models into character-level models. Their method resolves the prompt boundary problem, provides a more accurate character-level probability distribution, and achieves lower per-byte surprisal, indicating improved compression.
Key insights
- The 'covering' set C(σ) is finite for tokenizers adhering to strict-prefix monotonicity, allowing for the principled marginalization of token probabilities to derive character-level distributions.
- Practical experiments show that even with small beam sizes (e.g., K=8 to K=16), the method effectively captures most of the probability mass, indicating good performance-to-cost efficiency.
- Heuristic approaches like 'token healing' are empirically demonstrated to be significantly less accurate than the proposed probabilistic framework for resolving token-character mismatches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.