Reproduce “ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuqi Tang, Jing Yu, Zichang Su, Kehua Feng, Zhihui Zhu, Libin Wang, Lei Liang, Qiang Zhang and 2 more
ClinDEF, a dynamic evaluation framework from Zhejiang University and AntGroup, assesses large language models' clinical reasoning capabilities by generating unique patient cases and simulating interactive diagnostic dialogues, demonstrating that models consistently underperform in systematic information gathering and expressing diagnostic uncertainty.
Key insights
- Large language models consistently exhibit weaknesses in proactive and systematic information gathering, often adopting a passive or reactive approach in diagnostic dialogues.
- A significant safety concern is LLMs' pervasive overconfidence, as they frequently provide definitive diagnoses without adequately acknowledging or expressing diagnostic uncertainty, even with incomplete evidence.
- Achieving high diagnostic accuracy often involves a balanced strategy of comprehensive yet efficient information gathering, rather than merely more turns or fewer turns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.