Reproduce “PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aya E. Fouda, Abdelrahamn A. Hassan, Radwa J. Hanafy, Mohammed E. Fouda
PsychiatryBench introduces a multi-task benchmark for evaluating large language models (LLMs) in psychiatry, comprising 5,188 expert-annotated items from authoritative clinical sources across eleven distinct task types. The evaluation of 15 leading LLMs demonstrates a performance increase with newer models, with top performers achieving around 84% average scores, while also revealing persistent challenges in fine-grained clinical discrimination and multi-label classification.
Key insights
- LLMs show strong psychiatric knowledge and coherent clinical interpretations, but their capabilities are uneven across tasks, with substantial gaps identified in clinical consistency and safety for complex multi-turn follow-up and management.
- Domain-specialized models like MedGemma can rival generalist models in knowledge-intensive tasks, while top-tier generalists demonstrate superior performance in fluid, multi-step reasoning and cross-task consistency.
- A clear positive correlation exists between model release date and average performance, with 'thinking' variants (multi-step inference) providing architecture-dependent benefits, especially in demanding clinical judgment tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.