Reproduce “AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu, Qingjiang Shi

AM-Bench evaluates how missing or ambiguous task information affects tool-using agents by comparing complete tasks with versions that obscure goals, context, or constraints. It separately measures task completion and compliance, with success requiring both. In Constraint–Missing, agents completed 77.5% of tasks but achieved 0% success, compared with 76.5% success under Full; this applies to the tested model–harness grid.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.