Reproduce “BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, Eric Xu
BIGFINANCEBENCH establishes a workflow-grounded benchmark with 928 expert-authored tasks and derivation-level grading to evaluate financial-research agents on auditable analysis processes. Current top-performing AI systems achieved a rubric score of 58.8% and a final-answer accuracy of 44.3%, indicating considerable scope for improvement in accurately tracing financial conclusions.
Key insights
- Workflow-grounded rubrics provide diagnostic insights into agent performance, allowing for partial credit and localization of failures, which is distinct from mere final-answer accuracy.
- Current AI systems demonstrate substantial headroom for improvement in complex financial research, with the best model achieving only a 58.8% rubric score.
- The majority of agent failures occur in the initial 'Retrieval' and 'Definition' stages rather than in complex arithmetic, suggesting these steps are critical bottlenecks for financial agents.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.