Reproduce “Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Iryna Hartsock, Ghulam Rasool

Researchers at the H. Lee Moffitt Cancer Center & Research Institute provide a comprehensive review of recent advancements in Vision-Language Models (VLMs) specifically applied to medical report generation and visual question answering. The paper consolidates knowledge on VLM architectures, training strategies, and evaluation metrics, while also compiling details on 17 publicly available medical vision-language datasets and analyzing 15 notable medical VLM models.

Key insights

  • Transformer-based architectures are predominant in modern medical VLMs, often employing single-stream or dual-stream designs and encoder-decoder models for generative tasks.
  • Self-supervised learning paradigms, including Contrastive Learning, Masked Language Modeling, Masked Image Modeling, and Image-Text Matching, are standard for VLM pre-training in the medical domain.
  • While medical vision-language datasets are growing, their diversity and representativeness remain challenging, necessitating specialized evaluation metrics like RadGraph F1 and human expert assessment for clinical relevance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.