Reproduce “D-SCoRE: Document-Centric Segmentation and CoT Reasoning with Structured Export for QA-CoT Data Generation” on OpenResearch

About this paper

Weibo Zhou, Lingbo Li, Shangsong Liang

The D-SCoRE framework provides a training-free pipeline for generating diverse, reasoning-rich QA-Chain-of-Thought (CoT) datasets from raw text. It enables large language models fine-tuned on its synthetic data to achieve competitive or superior performance compared to human-annotated datasets, often with one-third the examples, by emphasizing implicit questions and quality control with a distinct critic model.

Key insights

• Prioritizing implicit, reasoning-intensive QA-CoT data significantly enhances LLM performance and facilitates reasoning transfer, even for downstream extractive tasks.
• D-SCoRE-generated datasets enable LLMs to achieve competitive or superior performance with substantially fewer training examples (approximately one-third) compared to human-annotated gold standards, demonstrating high data efficiency.
• Utilizing a heterogeneous pipeline with a specialized "Critic model" for quality control is crucial for preventing synthetic data errors, minimizing noise, and maximizing data fidelity, especially for reasoning-centric supervision.

Summary by alphaXiv.