Reproduce “ICBCBench: An Industry Consortium Benchmark for Financial Deep Research” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiya Li, Zhiwei Tang, Yizhou He, Chenghao Wang, Liang Feng, Xiao Sun, Dongrui Liu, Zichen Wen and 5 more
ICBCBench, an industry-consortium benchmark developed with over 50 financial experts, introduces a dual-track evaluation for Deep Research Agents (DRAs) in finance, assessing both objective factual accuracy and subjective long-form report quality across 180 tasks in English and Chinese. It demonstrates that LLM judges can achieve or exceed human expert consensus in subjective evaluation for complex financial analysis.
Key insights
- A significant performance gap exists between subjective report generation and precise, verifiable objective reasoning for current DRAs, suggesting the latter is more challenging.
- Open agentic frameworks generally outperform monolithic closed-source systems in orchestrating tool use, but their ultimate performance is still bottlenecked by the underlying base LLM.
- Some proprietary models exhibit an "illusion of competence," producing coherent, high-scoring subjective reports while demonstrating low factual accuracy in objective tasks.
- LLM judges, guided by structured rubrics, show strong alignment with human experts (Spearman's = 0.643) and can provide more stable and reliable subjective evaluations than individual human evaluators.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.