Reproduce “Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanhao Zou, Zhaozheng Yin

The AMiF framework from researchers at the University of Michigan and Stony Brook University enhances Medical Visual Question Answering by integrating unified multi-level modality alignment, hard negative mining with explicit discrimination, and selective knowledge fusion. It achieved state-of-the-art overall accuracy across VQA-RAD (80.49%), SLAKE (86.71%), PathVQA (65.33%), and VQA-2019 (83.20%) datasets, demonstrating improvements up to 5.29 percentage points over prior methods.

Key insights

  • A comprehensive, unified alignment strategy across multi-levels, multi-modalities, and multi-views significantly improves the robustness of medical vision-language representations.
  • Explicitly distinguishing hard negative samples through soft labels and a dedicated classification mechanism is crucial for learning highly discriminative features in specialized domains like medicine.
  • Selectively fusing only task-relevant prior knowledge, rather than indiscriminately incorporating broad knowledge bases, enhances Med-VQA performance by reducing noise and focusing on pertinent information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.