Reproduce “DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong and 2 more
DecisionPivot evaluates whether language models make correct typed decisions when decisive details change. It builds matched written cases with fixed options, groups related cases, and counts a group correct only when every member is correct. The benchmark includes 13 task families and 35 public test sets. Results separate some models with similar reference scores, but written-case tests do not establish end-to-end deployment performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.