Reproduce “Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Prafulla Kumar Choubey, Xin Su, Man Luo, Xiangyu Peng, Caiming Xiong, Tiep Le, Shachar Rosenman, Vasudev Lal and 4 more

Salesforce Research and collaborators developed a data synthesis pipeline, SynthKG, to create high-quality, ontology-free knowledge graphs from documents, then distilled this process into a smaller language model, Distill-SynthKG-8b. This distilled model achieved knowledge graph construction quality comparable to or exceeding larger LLMs at significantly lower inference cost, leading to enhanced retrieval-augmented generation performance.

Key insights

  • Decomposing knowledge graph construction into systematic steps (e.g., chunking, decontextualization, entity, and proposition extraction) significantly improves KG coverage and consistency compared to direct LLM prompting.
  • High-quality synthetic data generated by a multi-step teacher LLM can effectively transfer complex extraction capabilities to smaller, more efficient LLMs, enabling them to match or exceed the performance of larger, direct-prompted models.
  • Structured knowledge graphs, particularly those incorporating propositions, enable more effective and logically connected retrieval strategies, significantly enhancing downstream RAG performance and multi-hop reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.