Reproduce “Action Verification through Conjecture and Refutation for Language Model Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minhyeon Oh, Chiyeong Heo, Geonwoo Noh, Hoyoung Kim, Jangwon Son, Jungseul Ok
BlackSwan checks proposed state-changing actions by predicting the effect, listing testable suspicions, and testing them with observations or read-only probes before execution. Confirmed suspicions block the action; dismissed or unresolved ones do not. On AppWorld’s challenge split, it achieves 84.7% task goal completion versus 63.5% for reproduced ACCORD, with one demonstration and thinking on, at higher token and API costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.