Reproduce “Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi Sun, Kun Huang, Xiaocui Yang, Rong Tong, Kun Zhang, Soujanya Poria
The GenRDK framework introduces a method for Zero-shot Document-level Relation Triplet Extraction (ZeroDocRTE) by generating synthetic training data using large language models (LLMs) and applying a consistency-guided denoising strategy. This approach achieved F1-scores of 14.2 ± 1.3 on DocRED for ZeroDocRTE and 41.5 ± 8.7 on DocRED for ZeroDocRE, significantly outperforming a ChatGPT baseline by up to 19.6 F1.
Key insights
- Structured, multi-step prompting can effectively guide LLMs to generate complex, document-level synthetic data for information extraction tasks, overcoming challenges of simple few-shot prompting.
- Leveraging cross-document and multi-source consistency is crucial for mitigating LLM hallucination, substantially enhancing the quality and reliability of synthetically generated training data.
- Distilling knowledge from LLMs into a fine-tuned, specialized model using denoised synthetic data yields superior performance in zero-shot document-level relation extraction compared to direct LLM inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.