Reproduce “CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timothy Ossowski, Xinchi Liu, Danyal Maqbool, Vaibhav Dhanuka, Sheng Zhang, Hoifung Poon, Majid Afshar, Tyler Bradshaw and 1 more
CodeClinic introduces a benchmark for evaluating whether language-model agents can build, verify, and reuse executable clinical reasoning skills from guidelines and EHR data. On MIMIC-IV ICU tasks, reusable autoformalized functions improved compositional reasoning and reduced token use in many settings, but models still struggled with deeper clinical dependencies and maintaining accurate patient trajectories over time.
Key insights
- Local clinical decisions can appear accurate while complete longitudinal trajectories remain unreliable: Claude-Sonnet-4.6 reached 76.2% checkpoint action accuracy but only 34.0% full-trajectory accuracy.
- Reasoning quality declines as dependency depth increases; for Claude, accuracy dropped from 58.4% on Level 1 concepts to 41.1% on Level 3-plus concepts.
- Verified reusable functions generally reduce repeated schema exploration and improve deeper reasoning, but gains vary by model and can reverse for some configurations, such as the reported Gemma-4-31B surveillance result.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.