Reproduce “CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Patrick Huber, Armen Aghajanyan, Barlas Oğuz, Dmytro Okhonko, Wen-tau Yih, Sonal Gupta, Xilun Chen
Researchers at Meta AI introduce CCQA, a web-scale dataset of natural question-answer pairs extracted from Common Crawl using `schema.org` annotations, designed for in-domain pre-training of language models for Open-Domain Question Answering. Models pre-trained on CCQA demonstrate substantial performance improvements in zero-shot and low-resource settings across both generative and retrieval tasks.
Key insights
- Leveraging `schema.org` annotations enables the extraction of high-quality, natural question-answer pairs from web-scale data, addressing limitations of synthetic or small-scale natural datasets.
- In-domain pre-training on a large, natural QA corpus like CCQA substantially improves language model performance for Open-Domain Question Answering, particularly in zero-shot and low-resource settings.
- The rich, structured metadata within CCQA supports diverse future research, including answer selection, ranking, and multilingual QA, by providing more nuanced contextual information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.