Reproduce “Evaluating Large Language Models on Financial Report Summarization: An Empirical Study” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinqi Yang, Scott Zang, Yong Ren, Dingjie Peng, Zheng Wen

An empirical study evaluated GLM-4, Mistral-NeMo, and LLaMA3.1 for financial report summarization, introducing a new multi-lingual dataset and a novel LLM-based evaluation score to assess accuracy, informativeness, and coherence. LLaMA3.1 generally produced the highest quality summaries but was the slowest, while GLM-4 offered a good balance of performance and speed.

Key insights

  • Iruca-LLaMA3.1 consistently generated summaries with the highest accuracy, informativeness, and coherence across ROUGE and BERT metrics, although at a significantly slower processing speed.
  • Iruca-GLM-4 demonstrated a strong balance, performing well on semantic metrics and achieving the highest LLM Score Precision, while being considerably faster than LLaMA3.1.
  • The novel LLM Score provides a comprehensive, expert-like evaluation of summary quality, offering insights beyond traditional token-overlap or semantic similarity metrics in the financial domain.
  • The publicly released multi-lingual financial dataset serves as a crucial benchmark, fostering further research and development in domain-specific LLM applications for finance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.