Reproduce “Pre-Training Curriculum for Multi-Token Prediction in Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ansar Aynetdinov, Alan Akbik
This paper from Humboldt-Universität zu Berlin introduces curriculum learning strategies to enable Multi-Token Prediction (MTP) for smaller language models, demonstrating that a Forward curriculum improves both performance and inference speed (1.2-1.7x) for models with 1.3B and 3B parameters. The work also found that byte-level tokenization consistently yields better results for MTP in these smaller architectures.
Key insights
- Curriculum learning strategies, particularly the Forward curriculum, enable smaller language models (1.3B and 3B parameters) to effectively leverage Multi-Token Prediction (MTP), overcoming previous capacity limitations.
- The Forward curriculum balances improved performance and significant inference speed-ups (1.2-1.7x) via self-speculative decoding, while the Reverse curriculum improves LM performance and output quality but fails to provide speed benefits due to poor token acceptance rates.
- Byte-level tokenization consistently outperforms subword-level for MTP in smaller models, yielding better perplexity, benchmark scores, and output quality, likely due to simpler underlying dependencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.