Reproduce “MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Erfan Nourbakhsh, Ke Yang, Anthony Rios

MedProb evaluates medical vision-language models through frozen-representation probing rather than relying only on generated answers. Lightweight layer-wise classifiers consistently recover more accurate answers than prompting—up to 87.35% average accuracy and 89.94% macro-F1 in the reported comparisons—indicating that relevant medical information may be encoded internally but difficult to elicit reliably through text generation.

Key insights

  • Internal representations generally contain more linearly recoverable answer information than standard generation reveals: InternVL3-1B rises from 67.83% with prompting to 79.32% with probing, while Qwen3-VL-2B rises from 66.42% to 80.30%.
  • Probing narrows—but does not eliminate—the apparent advantage of larger models and is less sensitive to answer-option order; image blanking and shuffling also show that probes depend on visual evidence rather than text alone.
  • Medical adaptation does not reliably improve representation quality: general-purpose models outperform their medical counterparts in six of 14 matched pairs, although MediX-R1 provides notable gains over its base Qwen3-VL models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.