Reproduce “The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuexi Pan, Yongchao Chen

The authors introduce SciValidityBench, testing whether scientific readers use true records only when experimental state permits decisions. SciStateForge compiles source-linked dependencies from generated histories; readers choose actions scored against a TaskSpec oracle. Across 64 tasks, Qwen3-14B with typed SVL achieves a 0.981 valid-action rate versus 0.794 with Event Graph, in controlled tasks across four open model families.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.