Reproduce “EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yitong Qiao, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu, Kui Ren
EHR-Complex is a large-scale benchmark built on the full MIMIC-IV dataset, designed to evaluate medical agents on complex, interactive clinical reasoning tasks. It reveals that current LLM agents struggle significantly with population-level queries and medical-code grounding, with top models achieving around 62% accuracy.
Key insights
- Current LLM agents demonstrate a substantial performance gap between patient-level and population-level clinical reasoning, indicating difficulties with cohort-level aggregation and complex temporal alignment.
- Failure modes are predominantly SQL logic errors, medical-code lookup errors (mapping clinical terms to database identifiers), and semantic misunderstanding of clinical requests.
- Execution-guided EHR reasoning is stochastically fragile; even with a temperature of 0, minor variations in agent trajectories can lead to plausible but incorrect answers, especially in complex population-level tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.