Reproduce “Certified Selective Automation of LLM Agent Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chengguang Gan, Yunhao Liang, Qinghao Zhang, Shiwen Ni
TASKBOOT certifies usable automation of agent review by resampling whole tasks rather than individual trajectories to account for correlated attempts. It estimates error among auto-rejected trajectories and selects the highest-coverage threshold within a chosen error budget. On WEB-A, it certifies .585 reject coverage for a trained 4B judge, versus .510 for its base version; this is not a finite-sample guarantee.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.