Reproduce “Fewer Truncations Improve Language Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, Stefano Soatto
This AWS AI Labs paper introduces Best-fit Packing, a data preparation method for large language models that minimizes document truncation during pre-training. Models trained with this approach demonstrate improved performance across various natural language and programming tasks, alongside a substantial reduction in closed-domain hallucination, without compromising training efficiency.
Key insights
- The standard 'concatenate-then-split' data preparation method for LLMs, while token-efficient, introduces arbitrary truncations that compromise document integrity and hinder language model performance.
- Preserving complete document contexts during pre-training, especially for documents shorter than the context window, fundamentally enhances a model's ability to learn coherent relationships and reduces hallucination.
- Efficient combinatorial packing algorithms, like an optimized Best-Fit Decreasing approach, can achieve high training efficiency comparable to concatenation while significantly reducing truncations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.