Reproduce “Subgroup Validity in Machine Learning for Echocardiogram Data” on OpenResearch

About this paper

Cynthia Feeney, Shane Williams, Benjamin S. Wessler, Michael C. Hughes

Researchers at Tufts University and Tufts Medical Center investigated subgroup validity in machine learning models for echocardiogram data. Their work highlighted pervasive underreporting of granular sociodemographic data in open datasets, demonstrating that current data limitations prevent statistically robust assessments of model performance across diverse racial and ethnic subgroups.

Key insights

• Open echocardiogram datasets widely used for machine learning lack comprehensive sociodemographic reporting and exhibit severe imbalances, with white patients consistently overrepresented and gender-diverse groups entirely unrepresented.
• The inherent limitations of current data, including small sample sizes for racial and ethnic minorities, make statistically robust and definitive assessments of machine learning model subgroup validity impossible.
• Despite observed performance variations across patient subgroups in exploratory analyses, the wide confidence intervals for underrepresented groups mean that definitive conclusions about model fairness cannot be drawn with current data.

Summary by alphaXiv.