Reproduce “MedFM-Robust: Benchmarking Robustness of Medical Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiangxiang Cui, Tianjin Huang, Yifang Wang, Lijie Hu, Lu Yin

This paper introduces MedFM-Robust, a comprehensive benchmark to evaluate the robustness of medical foundation models, encompassing vision-language and segmentation tasks, under 40 domain-specific perturbations across eight medical imaging modalities. The research quantifies performance degradation, showing how fine-tuning strategies critically influence robustness and that medical-specific corruptions significantly challenge model reliability.

Key insights

  • A performance-robustness trade-off exists in segmentation models, where full fine-tuning generally offers the highest robustness, while parameter-efficient methods like LoRA can lead to substantial degradation under perturbations.
  • Medical-specific corruptions, such as OCT Motion Artifacts and Light Reflection, are primary drivers of failure, accounting for 9 out of the top 15 most challenging perturbations for segmentation models, highlighting the need for domain-specific evaluations.
  • Robustness varies significantly by task and model type: fine-tuning for specialized capabilities (e.g., visual grounding) can enhance performance but compromise robustness, whereas zero-shot inference (e.g., captioning) often preserves pretrained stability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.