Reproduce “BENCHCOMPASS: From Scores to Signals for Training and Harness Decisions in Payment-Domain LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sijie Dong, Wei Ren, Xuanwei Hu, Jiawei Luo, Zifan Wang, Xiaoyun Feng, Hui Cai, Lyuxin Xue and 4 more
BenchCompass evaluates models across rule knowledge, evidence-grounded reasoning, and answer-preserving task-input attacks. It tests decisions without supplied rules, with evidence, and with evidence plus misleading input changes. Gemini 3.1 Pro Preview scores 14.6% without evidence and 89.6% with it on the reasoning subset; it scores 81.7% on a separate 115-case attack set. These are accuracies, not a paired attack degradation estimate.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.