Reproduce “MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nasir Hayat, Krzysztof J. Geras, Farah E. Shamout

MedFuse, a multi-modal fusion architecture from NYU Abu Dhabi and NYU Grossman School of Medicine, integrates clinical time-series data and chest X-ray images, specifically addressing partially paired data common in healthcare settings. The LSTM-based model processes uni-modal representations sequentially, achieving improved AUROC and AUPRC over uni-modal baselines and other fusion methods for in-hospital mortality prediction and phenotype classification, particularly benefiting mixed and chronic conditions like pneumonia.

Key insights

  • MedFuse effectively leverages a relatively small percentage (10-20%) of uni-modal samples during fine-tuning for optimal performance, demonstrating robustness to data incompleteness.
  • The sequential processing of uni-modal representations by an LSTM module provides a flexible and scalable way to integrate diverse clinical data without assuming strong inter-modality correlations.
  • Integrating chest X-ray images with clinical time-series data yields more substantial performance gains for mixed and chronic conditions (e.g., pneumonia) than for acute conditions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.