Reproduce “Literature-Grounded Novelty Assessment of Scientific Ideas” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Simra Shahid, Marissa Radensky, Raymond Fok, Pao Siangliulue, Daniel S. Weld, Tom Hope
Researchers from Microsoft, the University of Washington, and the Allen Institute for AI developed the Idea Novelty Checker, an automated system that assesses the novelty of scientific ideas by employing a two-stage retrieval-then-rerank approach and generating literature-grounded rationales. The system achieved a Cohen's Kappa of 0.59 in agreement with expert judgments, outperforming existing LLM-based novelty checkers, and demonstrated 89.66% accuracy in identifying non-novel ideas.
Key insights
- Reliable novelty assessment requires a standardized, facet-based definition of novelty and a robust mechanism for identifying comprehensively relevant prior work, as highlighted by formative expert studies.
- Leveraging expert-annotated examples for in-context learning significantly enhances an LLM's ability to classify novelty accurately and provide grounded reasoning.
- A two-stage retrieval and re-ranking pipeline, specifically incorporating facet-based LLM re-ranking, is critical for surfacing the most pertinent papers and enabling accurate "not novel" classifications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.