Reproduce “Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanyi Su, Yu Zeng, Hongshuai Wang, Chenyan Wang, Zhifeng Gao, Xiaohui Yang
The authors test source-label sensitivity by defining chemical targets through edits to source molecules. For Claude Opus 4.8 on BACE classification, 91.2% of paired predictions switched toward a replacement source label, versus 36.0% for DeepSeek V4 Pro Preview. This is a directional-response measure, not accuracy, and does not establish the sole cause of errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.