Reproduce “ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jun Xiang, Zhijie Bao, Rong Hu, Kaizhou Qin, Wei Chen, Zhongyu Wei
ObGynLongBench evaluates whether language models can make obstetric and gynecologic decisions from complete longitudinal EHRs rather than preselected evidence. Across 17 models and 1,500 decision cases, performance consistently fell when models had to retrieve and temporally integrate evidence from raw records: Gemini-3-Flash dropped from 79.1% with localized evidence to 68.7% with full histories, while an active-search strategy improved average history-level accuracy from 59.6% to 64.2%.
Key insights
- Raw longitudinal EHR reasoning is substantially harder than applying a clinical rule to already-localized evidence; history-level accuracy decreased from 61.0% on simple L1 cases to 54.2% on long-range L3 cases.
- Longer records reduce both accuracy and substantive use of the relevant supporting evidence, while recency weighting can omit older findings needed for complex decisions.
- Errors are patient-linked: mistakes on earlier decision points predict later mistakes in the same history, suggesting persistent difficulty with particular patient records rather than independent question failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.