Reproduce “Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sina J. Semnani, Jirayu Burapacheep, Arpandeep Khatua, Thanawan Atchariyachanvanit, Zheng Wang, Monica S. Lam
Researchers from Stanford University introduced and formalized Corpus-Level Inconsistency Detection (CLID), developing CLAIRE, an LLM-powered assistant, to identify naturally occurring contradictions within large knowledge corpora like Wikipedia. Their system enabled human editors to detect 64.7% more inconsistencies per hour and revealed that at least 3.3% of English Wikipedia facts contain contradictions.
Key insights
- Corpus-level inconsistencies are prevalent in Wikipedia, with at least 3.3% of facts containing contradictions, challenging its assumed consistency.
- LLMs can be effectively leveraged as agentic assistants to enhance human curation and improve the accuracy of the very knowledge corpora they are trained on.
- The presence of inconsistencies in Wikipedia impacts the reliability of existing NLP benchmarks (e.g., AmbigQA, FEVEROUS) that assume a singular, consistent source of truth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.