Reproduce “LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It” on OpenResearch

About this paper

Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris

This research quantifies the "omission blindness" of large language model (LLM) judges in identifying missing information within AI-generated clinical notes. The study found that while traditional LLM judges are effective at detecting added or altered content, they struggle with omissions, which are the most common errors. By restructuring the evaluation task into explicit "presence checks," the work achieved substantial improvements, with an enumerate-then-check pipeline detecting 24.6% of omissions at a 2.7% false-alarm rate and an evolved prompt reaching 36.9% detection at a 6.2% false-alarm rate on the new OmissionBench benchmark.

Key insights

• LLM judges exhibit a consistent and profound "presence/absence asymmetry," reliably detecting commissions (added/altered content) but performing at near-chance levels for semantic omissions.
• Conventional prompt engineering, output format changes, and automated prompt optimization do not overcome this fundamental "omission blindness" in LLM judges.
• Restructuring the omission detection task into discrete, verifiable "presence checks" of extracted facts, rather than open-ended 'is anything missing?' queries, effectively recovers omission detection capability.

Summary by alphaXiv.