Reproduce “LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zitong Xu, Huiyu Duan, Bingnan Liu, Guangji Ma, Jiarui Wang, Liu Yang, Shiqi Gao, Xiaoyu Wang and 4 more

This research introduces EBench-18K, a comprehensive large-scale benchmark for Text-guided Image Editing (TIE) models, featuring 18,360 edited images and extensive human preference annotations across perceptual quality, editing alignment, and attribute preservation. It also presents LMM4Edit, an LMM-based automatic evaluation metric that accurately predicts fine-grained scores and answers task-specific questions, demonstrating strong alignment with human perception.

Key insights

  • TIE models frequently face challenges in achieving a balanced performance across key evaluation dimensions: perceptual quality, how well edits align with instructions, and the preservation of unedited image regions.
  • Comprehensive human annotations, specifically multi-dimensional Mean Opinion Scores (MOSs) and task-specific question-answering, are essential for accurately assessing TIE model performance and understanding their limitations.
  • Fine-tuning Large Multimodal Models (LMMs) with specialized instruction tuning and adaptive low-rank adaptation (AdaLoRA) significantly enhances their ability to align with human perception for complex, multi-dimensional image editing evaluation tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.