Reproduce “FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Neng Wang, Hongyang Yang, Christina Dan Wang
A comprehensive instruction tuning paradigm benchmarks diverse open-source large language models (LLMs) in financial datasets, revealing distinct performance profiles across task-specific, multi-task, and zero-shot scenarios for a total estimated training cost of approximately $302.4. This work provides an open and cost-effective framework to adapt LLMs for specialized financial applications.
Key insights
- Different open-source LLMs exhibit varying inherent strengths and weaknesses across specific financial NLP tasks, with models like BLOOM excelling in information extraction while others like MPT perform better in sentiment analysis.
- Multi-task instruction tuning can improve performance on information extraction tasks (e.g., Relation Extraction) but may lead to slight performance declines or interference on classification tasks.
- Zero-shot instruction tuning highlights distinct generalization capabilities; ChatGLM2 and Falcon demonstrated strong zero-shot performance, effectively understanding unseen instructions for novel financial scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.