Reproduce “H-DDx: A Hierarchical Evaluation Framework for Differential Diagnosis” on OpenResearch

About this paper

Seungseop Lim, Gibaeg Kim, Hyunkyung Lee, Wooseok Han, Jean Seo, Jaehyo Yoo, Eunho Yang

AITRICS and KAIST researchers developed H-DDx, a hierarchical evaluation framework for large language models (LLMs) in differential diagnosis, which leverages the ICD-10 taxonomy and a new metric, HDF1. This framework provides a more clinically nuanced assessment, reordering model rankings and highlighting the strengths of domain-specialized LLMs previously underestimated by flat evaluation metrics.

Key insights

• Conventional flat metrics misrepresent LLM diagnostic capabilities by severely penalizing clinically plausible near-misses, leading to an undervaluation of certain models, particularly domain-specialized ones.
• HDF1 reorders model rankings, demonstrating its ability to recognize clinically relevant diagnostic coherence; for example, MediPhi rose 18 ranks from 20th (Top-5 Accuracy) to 2nd (HDF1).
• The framework provides enhanced interpretability by revealing a universal 'hierarchical cascade pattern' where LLM performance consistently degrades with increasing diagnostic specificity but often retains broader clinical context.

Summary by alphaXiv.