Reproduce “Analysing The Impact of Sequence Composition on Language Model Pre-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miłoś, Yuxiang Wu, Pasquale Minervini
Researchers from the University of Edinburgh, Xiaomi AI Lab, and the University of Warsaw systematically analyzed how pre-training sequence composition affects language model performance. They found that strategies like Intra-Document Causal Masking (INTRADoc) and retrieval-based BM25Chunk improved language modeling ability, in-context learning, and knowledge memorization, with INTRADoc showing up to a 37.5% Exact Match improvement in knowledge memorization for 8K context models.
Key insights
- Cross-document distractions within pre-training sequences significantly impede a language model's fundamental language modeling ability and its capacity for in-context learning and knowledge memorization.
- Even under standard causal masking, increasing the semantic coherence or 'burstiness' of documents within a pre-training chunk, as with BM25Chunk, substantially mitigates distraction and improves performance.
- Explicitly enforcing document boundaries during causal masking (INTRADoc) provides a more effective learning signal, leading to superior knowledge acquisition and context utilization, even challenging previous assumptions about packing benefits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.