Reproduce “Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qinan Zhang, Xinyu Wang, Qihang Jin, Tianyuan Wang, Hongbing Luo, Pengfei Li, Wenjun Lv, Yu Kang and 1 more
The authors introduce GridWM-Judge, an audit that tests whether vision-language judges keep verdicts across presentation changes and respond to interventions that change task outcomes. It evaluates deterministic MiniGrid tasks using simulator labels and rendered evidence. In expanded reference-aided Task D, GPT-5.4’s raw IDR was 0.322 versus Claude Sonnet 4.6’s 0.088 across 432 groups; these results are not a universal ranking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.