Reproduce “Joint Optimization of Tokenization and Downstream Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki
A general-purpose method is introduced for jointly optimizing tokenization with any downstream NLP model, using only the downstream loss values to guide the tokenizer's learning. The approach demonstrates consistent performance improvements over existing methods across text classification and machine translation tasks while maintaining consistency between training and inference phases.
Key insights
- Optimal tokenization is task-specific, and jointly optimizing it with the downstream task consistently improves performance across diverse NLP applications.
- Maintaining consistency between training and inference phases for the downstream model is crucial for effective joint optimization, addressing a key limitation of prior methods.
- Source-side and target-side tokenization in machine translation may benefit from different granularities, with source often favoring finer-grained segmentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.