Reproduce “CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rishi Srivastava
CFAgentBench provides a reproducible environment and benchmark to evaluate autonomous language model agents performing real-world construction finance operations. It features an executable multi-system environment and demonstrates that single-attempt accuracy significantly overstates agent reliability in this complex, high-stakes domain, particularly for tasks involving guarded financial transactions.
Key insights
- A substantial drop occurs from `pass_1` (single-attempt accuracy) to `pass_k` (reliability across multiple runs), indicating that initial successes significantly overstate an agent's deployable competence.
- Agent performance is highly heterogeneous across different construction-finance domains, with models excelling in some areas (e.g., Billing) while struggling severely in others (e.g., Project Accounting).
- Tasks deemed 'difficulty-1' (human-quick) proved unexpectedly challenging for agents, with zero `pass_5` scores on critical tasks like COI payment holds, suggesting a mismatch between human and agent perceived difficulty.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.