Reproduce “ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research” on OpenResearch

About this paper

Junyong Lin, Lu Dai, Ruiqian Han, Yijie Sui, Ruilin Wang, Xingliang Sun, Qinglin Wu, Min Feng and 2 more

ScIRGen introduces a comprehensive framework for synthesizing large-scale, realistic Retrieval-Augmented Generation (RAG) datasets tailored for scientific research. Demonstrated with ScIRGen-Geo for geoscience, the framework produced a dataset of 61,376 QA pairs that achieves a cognitive diversity index of 0.91, significantly outperforming existing benchmarks in evaluating higher-order reasoning capabilities.

Key insights

• Augmenting dataset representations with rich, paper-extracted context significantly improves retrieval performance across various models, particularly those capable of processing longer input contexts.
• Designing datasets with high cognitive diversity, as measured by Bloom's Taxonomy, is crucial for comprehensively evaluating the full spectrum of reasoning abilities required in scientific inquiry.
• The SePer-Filter, based on an LLM's 'belief shift' in response to context, offers a more reliable and cost-effective method for ensuring the quality of synthetically generated data compared to LLM-as-a-judge approaches.
• State-of-the-art LLMs consistently struggle with complex causal reasoning and higher-order cognitive tasks (e.g., Applying, Analyzing) within scientific contexts, even with enhanced contextual information.

Summary by alphaXiv.