Reproduce “MSDiagnosis: A Benchmark for Evaluating Large Language Models in Multi-Step Clinical Diagnosis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruihui Hou, Shencheng Chen, Yongqi Fan, Guangya Yu, Lifeng Zhu, Jing Sun, Jingping Liu, Tong Ruan
Researchers from East China University of Science and Technology and Ruijin Hospital developed MSDiAGNOSIS, a benchmark for evaluating large language models in multi-step clinical diagnosis, addressing limitations of single-step tasks and traditional NLP metrics. The benchmark, along with a new multi-stage reasoning framework, revealed that current LLMs perform significantly below human experts in complex diagnostic tasks, though the framework improves diagnostic accuracy and interpretability.
Key insights
- Large Language Models currently exhibit poor performance on multi-step clinical diagnosis, showing a substantial gap compared to human diagnostic capabilities, particularly in differential diagnosis.
- The proposed multi-stage framework, incorporating forward inference, backward inference, reflection, and refinement, significantly enhances LLM diagnostic accuracy and the interpretability of generated criteria.
- The multi-step diagnostic process is highly interdependent, with the primary diagnosis stage critically influencing the accuracy and interpretability of the final diagnosis, indicating error propagation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.