Reproduce “WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin

The WILDVISION project introduces a human-preference-driven framework for evaluating Vision-Language Models in real-world scenarios, consisting of a live interaction platform (WILDVISION-ARENA) and a human-aligned automated benchmark (WILDVISION-BENCH). This framework reveals that GPT-4o leads in human preferences and demonstrates a 0.94 correlation between the automated benchmark and human judgments.

Key insights

  • Human-centered evaluation, through direct user preferences, offers a more practical and meaningful assessment of VLM utility than traditional static benchmarks.
  • Dynamic and continuous evaluation frameworks are crucial for accurately tracking and understanding performance shifts in rapidly evolving VLM models.
  • Extensive analysis of real-world interactions reveals persistent VLM limitations in areas like complex text recognition, spatial reasoning, and mitigating hallucination.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.