Reproduce “TREX: Tokenizer Regression for Optimal Data Mixture” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Inho Won, Hangyeol Yoo, Minkyung Cho, Jungyeul Park, Hoyun Song, KyungTae Lim
TREX, developed by researchers from KAIST and Upstage AI, proposes a regression-based framework for optimizing multilingual data mixtures in tokenizer training. The method significantly improves compression efficiency by up to 10.21% compared to heuristic baselines and reduces large language model training costs by over 200 hours through efficient proxy tokenizer-based prediction.
Key insights
- The 'Rank Invariance' principle holds: the relative ranking of data mixture efficiencies remains consistent across varying corpus and vocabulary scales (e.g., 1GB-64K to 30GB-200K, ρ ".974").
- An 'efficient bias' in language proportions, rather than strict uniformity, optimally biases the tokenizer for overall compression and specific script-level gains, particularly for non-Latin languages.
- Optimizing tokenizer compression at the pre-training stage directly translates to substantial computational cost reductions and improved training efficiency for large language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.