Reproduce “FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxuan Lu, Kong Wang, Yemin Wang, Qingmei Tang, Hongwei Zeng, Xiang Chen, Jiahao Pi, Shujian Deng and 4 more
Researchers from Shanghai AI Laboratory and collaborating institutions developed FinToolBench, a runnable benchmark to evaluate large language model agents interacting with real-world financial tools. It assesses both an agent's task capability and its adherence to finance-specific compliance constraints, finding that current models exhibit trade-offs between these aspects while explicit attribute injection improves compliance.
Key insights
- LLM agents exhibit a clear trade-off between achieving high task capability (e.g., tool execution success, answer quality) and maintaining finance-specific compliance, with no single model excelling at both simultaneously.
- Explicitly providing finance attributes to the LLM planner significantly improves conditional execution reliability and substantially reduces compliance mismatch rates for timeliness, intent, and domain.
- The benchmark's auditable tool traces and diagnostic metrics offer detailed insights into agent failures, distinguishing between functional errors and critical compliance violations in financial scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.