Reproduce “SciFact-Open: Towards open-domain scientific claim verification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, Hannaneh Hajishirzi
SCIFACT-OPEN introduces a benchmark for open-domain scientific claim verification against 500,000 abstracts, revealing that state-of-the-art models suffer a substantial F1 score degradation of 15-30 points compared to constrained environments. The work also characterizes phenomena like uneven evidence distribution, specificity mismatches, and conflicting evidence in large scientific corpora.
Key insights
- Current state-of-the-art scientific claim verification models struggle significantly to generalize, showing substantial performance drops when moving from constrained datasets to open-domain environments.
- The negative sampling ratio used during model training critically influences open-domain performance, dictating the trade-off between precision and recall.
- Evidence in open-domain scientific literature is characterized by uneven distribution, frequent specificity mismatches between claims and supporting evidence, and the common occurrence of conflicting information for a single claim.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.