Reproduce “Disentangling Task Difficulty from Run-Level Failure in Agent Failure Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohsen EsfandyariDoulabi, Lawrence Arkoh, Biruk Tadesse, Vaishvi Patel, Mehul Sharma, Marcelo d'Amorim, Wesley Assunção
The authors evaluate whether failure scores identify failing agent runs or mainly distinguish model–task difficulty. They compare pooled AUROC across runs with within-task AUROC among repeated attempts of the same model and task. On C1, a difficulty oracle scores 0.9454 pooled and 0.5000 within-task; early run-level signal is weak, but rises to 0.598 after ten turns in a fixed cohort.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.