Reproduce “ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxing Lu, Yushuhong Lin, Wenqi Shi, J. Ben Tamo, Xukai Zhao, Jinzhuo Wang, May Dongmei Wang
A collaborative team developed CLINENV, an interactive multi-stage environment derived from real Electronic Health Records (EHRs), to evaluate large language models (LLMs) in complex, longitudinal inpatient care simulations. LLMs achieved a peak decision F1 score of 0.306, demonstrating significant challenges in multi-stage clinical reasoning, medication selection, and efficient information synthesis compared to human-level performance.
Key insights
- Current LLMs struggle significantly with complex, multi-stage clinical reasoning, showing a sharp decline in performance on long-horizon cases and later stages of patient management, even when information is available.
- There is a notable decoupling between outcome quality and process efficiency; some models achieve similar decision scores but with highly inefficient information gathering and high resource waste.
- LLMs demonstrate better performance in diagnostic decisions (F1 ~0.40) compared to management decisions (F1 ~0.13), with particular difficulty in selecting precise medication identities rather than just action types.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.