Reproduce “High-Stakes Decisions with Language Models: Insights from Emergency Triage” on OpenResearch

About this paper

Khurram Yamin, Christopher Kelly, Bryan Wilder, Eric Horvitz

An investigation into language model behavior during emergency triage reveals that critical decision failures, like under-triage, often arise from implicit, misaligned utility functions rather than solely from diagnostic errors. This work demonstrates that explicit, plain-language utility instructions can steer capable LLMs to align their recommendations with desired safety-resource trade-offs, offering a method for more controlled and transparent decision support.

Key insights

• LLM decision failures in high-stakes settings can be driven by implicit utility functions that prioritize certain outcomes (e.g., resource conservation) over others (e.g., patient safety), rather than solely by poor diagnostic inference.
• Frontier LLMs possess high discriminatory power to rank risks (AUROC 0.95–0.99), indicating strong underlying predictive capability, even if their default decision thresholds or calibration are misaligned.
• Explicitly defining error cost ratios through plain-language prompts allows for effective steering of capable LLMs, enabling them to adjust their recommendations to desired safety-resource trade-offs, which is crucial for aligning AI with human values.

Summary by alphaXiv.