Reproduce “EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ron Yosef, Moran Yanuka, Yonatan Bitton, Dani Lischinski
Researchers from The Hebrew University of Jerusalem, Tel Aviv University, and Google Research introduced EditInspector, a comprehensive human evaluation benchmark for text-guided image edits. The work revealed that state-of-the-art Vision and Language Models (VLMs) often struggle with multi-dimensional edit assessment and frequently hallucinate, while their proposed methods and a fine-tuned LLaVA model achieve competitive or superior performance in specific evaluation tasks.
Key insights
- State-of-the-art VLMs generally perform suboptimally as comprehensive edit inspectors, demonstrating significant weaknesses in classifying inaccurate edits, detecting artifacts, understanding contextual consistency, and handling 'Remove' edit types.
- VLMs exhibit high hallucination rates when generating detailed difference captions, with even the best models achieving only 12% accuracy in detecting all differences.
- Specialized computational methods and efficient fine-tuned models, supported by robust data augmentation, can outperform or achieve competitive performance against large, general-purpose VLMs for specific edit evaluation tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.