Reproduce “Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level III” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pranam Shetty, Abhisek Upadhayaya, Parth Mitesh Shah, Srikanth Jagabathula, Shilpi Nayak, Anna Joo Fee
A comprehensive evaluation of 23 large language models on mock CFA Level III exams reveals that top-performing frontier models, including o4-mini, Gemini 2.5 Pro, and Claude Opus 4, achieve professional-grade performance by exceeding the estimated 63% passing threshold. The study also highlights the effectiveness of Chain-of-Thought prompting and identifies systematic differences between human and LLM-based essay grading.
Key insights
- Human graders consistently awarded higher scores than the LLM-grader (GPT-4.1) for essay responses, showing an average difference of 5.6 points, indicating systematic discrepancies in grading leniency and strictness.
- Chain-of-Thought with Self-Consistency (CoT-SC) significantly improved LLM performance, yielding a 7.8 percentage point increase in MCQ accuracy and the highest mean essay scores compared to Zero-Shot.
- Advanced prompting strategies, while enhancing accuracy, incurred substantial increases in computational cost (up to 11.1x) and processing time (up to 10.5x), highlighting a critical accuracy-efficiency trade-off for deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.