Reproduce “WRAP++: Web discoveRy Amplified Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiang Zhou, Yunhao Wang, Xing Wu, Tinghao Yu, Feng Zhang
WRAP++ creates synthetic pretraining data by pairing hyperlinked Wikipedia pages and asking an instruction-tuned generator to produce joint question–answer examples whose answers depend on both documents. It uses dual-link and co-mention graph motifs to select related pages. On OLMo-3-7B, it reaches 49.13% SimpleQA pass@128 versus 43.69% for Extended WRAP after one epoch, on Wikipedia, with unequal data volumes.
Key insights
- Leveraging human-curated web hyperlinks as topological signals allows for the discovery of high-confidence semantic relationships between documents, vital for quality cross-document data.
- Synthesizing joint QA that explicitly demands reasoning across multiple documents fosters the internalization of complex, multi-hop knowledge structures by LLMs.
- The combinatorial growth enabled by cross-document relation discovery offers a scalable solution to the data saturation challenges faced by single-document synthetic data generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.