Reproduce “VLTest: Finding Joint Image-Text Failures in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bin Duan, Jintao Lin, Guowei Yang
VLTest tests vision–language models for failures under paired image and text changes. It compares a reconstructed seed with image-only, text-only, and joint inputs; a VLM judge verifies answer preservation, and both single-change controls must pass for attribution. On PaliGemma VQA, it finds 196.8 attributed joint failures per 1,000 calls versus 155.9 for EXT, not proof of non-additive interaction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.