Reproduce “Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gabriele Prato, Shagun Sodhani, Alessandro Sordoni, Sarath Chandar
The study from Mila, Meta (FAIR), and Microsoft Research explores how document packing strategies during Large Language Model pre-training influence latent multi-hop reasoning, finding that dynamic packing with cross-document attention enhances recall precision and reduces hallucinations, with performance peaking at an optimal packing density.
Key insights
- Document packing fundamentally impacts LLM knowledge organization for complex reasoning, moving beyond its role as a mere computational efficiency technique.
- Cross-document attention within packed sequences is indispensable for enhancing latent multi-hop reasoning capabilities; its absence negates packing's benefits.
- Dynamic repacking of documents at each training epoch is crucial for performance, preventing models from forming rigid associations and fostering diverse contextual learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.