Reproduce “MMDM: Measuring Massive Multitask Decision Making” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhao Shen, Qing Zong, Yue Guo, Zhanqiu Zhang, Yiwen Guo, Hong Cheng
MMDM introduces a benchmark for testing decision models on long-context choices, interdependent options, and constraints. It covers single-select, complete multi-select, and probability-distribution tasks across 5,448 questions. Perplexity Decider-V1.2 scores 78.75% on World Knowledge but 47.95% on Constraint Planning; the strongest hard multi-select accuracy is 36.81%. These category comparisons are not causal, and the benchmark tests fixed text rather than execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.