Reproduce “Evidence-Informed LLM Beliefs for Continual Scientific Discovery” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dhruv Agarwal, Reece Adamson, Andrew McCallum, Peter Clark, Ashish Sabharwal, Bodhisattwa Prasad Majumder
This research enables Large Language Models (LLMs) to continually update their beliefs based on prior discoveries during scientific exploration, addressing the limitations of static surprise metrics. The approach integrates evidence-informed prior beliefs and enhanced search strategies, leading to a 30.63% increase in the yield of genuinely novel discoveries.
Key insights
- Human expert studies confirm that prior discoveries inform the perceived novelty of new hypotheses, highlighting the flaw in static surprisal metrics for continual discovery.
- Dynamically updating an LLM's prior beliefs through in-context learning (specifically top-k retrieval) effectively reduces spurious surprisals by aligning prior beliefs with eventual posteriors.
- Merely changing the scalar reward for search is insufficient; explicit search mechanisms like belief-update filtering and diversity maximization are essential for steering LLMs towards genuinely novel hypotheses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.