Reproduce “zip2zip: Inference-Time Adaptive Tokenization via Online Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Saibo Geng, Nathan Ranchin, Yunzhen yao, Maxime Peyrard, Chris Wendler, Michael Gastpar, Robert West

Researchers from EPFL introduced `zip2zip`, an adaptive tokenization framework that dynamically adjusts a Large Language Model's vocabulary during inference using online compression. The method reduces sequence length by 13-41% across various domains and yields over 100% prefill and 46% decode speedups on NVIDIA H100 GPUs by creating and embedding new 'hypertokens' on the fly.

Key insights

  • Lossless online compression (LZW) can effectively form semantically meaningful 'hypertokens' on-the-fly, allowing for dynamic vocabulary adaptation that significantly reduces token sequence length.
  • A dedicated, lightweight hyper-encoder can compute embeddings for new dynamic tokens, allowing existing transformer architectures to process and generate from an expanded, adaptive vocabulary without major architectural overhauls.
  • Training with an auxiliary reconstruction loss ensures that dynamic token embeddings are information-preserving, making the compression effective while maintaining the model's language understanding and generation capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.