Reproduce “Beyond Classification: Financial Reasoning in State-of-the-Art Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, Sol Jin
An investigation into language models' capacity for generating financial investment opinions reveals that reasoning capabilities emerge in models from 6 billion parameters upwards, with an In-Context Question Answering method proving effective for producing controlled and human-preferred outputs.
Key insights
- The ability to generate coherent financial reasoning emerges in language models at parameter scales around 6 billion and improves with increasing model size.
- Instruction tuning consistently improves model performance, yet model size appears to be a more critical determinant for coherent financial reasoning than the fine-tuning dataset size.
- The proposed In-Context Question Answering (ICQA) method significantly enhances controlled text generation and is preferred by human evaluators over full-text inputs and even expert-written theses.
- Generic LLM-based evaluators (e.g., GPT-4) exhibit low agreement with human judgments for complex financial reasoning tasks, indicating their unsuitability without domain-specific training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.