Reproduce “Lossless Vocabulary Reduction for Auto-Regressive Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida, Shin'ya Yamaguchi, Tomoya Ohba, Tamao Sakao, Susumu Takeuchi

Researchers at NTT Computer and Data Science Laboratories developed a theoretical framework and efficient algorithms for lossless vocabulary reduction in auto-regressive language models. The K-LVR method enables models to cooperate by converting their next-token distributions to an arbitrary shared sub-vocabulary without accuracy loss, outperforming heuristic baselines and offering improved inference efficiency over byte-level approaches.

Key insights

  • Lossless vocabulary reduction to any arbitrary sub-vocabulary is theoretically possible, ensuring the transformed model generates text with the exact same probability distribution as the original.
  • Employing sub-vocabularies that contain multi-byte tokens significantly improves inference efficiency during model cooperation compared to strictly byte-level operations.
  • A Top-K approximation (K-LVR) offers a practical and accurate method for vocabulary reduction, achieving comparable performance to full reduction while bounding computational overhead.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.