Reproduce “Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi
The authors test whether wording changes alter triage recommendations, comparing five language models with physicians across 6,232 scenarios. The benchmark includes baseline cases and controlled gender and tone rewrites. GPT-4o has 91.7% baseline accuracy versus physicians’ 90.5%, but drops 10.11 percentage points under colorful tone. Degradation persists when physician-majority recommendations remain stable, so baseline accuracy alone does not establish robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.