Reproduce “How Well Can Vision Language Models See Image Details?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenhui Gou, Abdulwahab Felemban, Faizan Farooq Khan, Deyao Zhu, Jianfei Cai, Hamid Rezatofighi, Mohamed Elhoseiny
Researchers developed a method to assess and boost Vision-Language Models' capacity for perceiving image details, revealing that existing models struggle with pixel-level information. Their proposed Pixel Autoencoded Large MultiModal Model (PAE-LaMM), trained with a pixel value prediction task, achieved over 10 cIoU points average improvement in referring image segmentation and 70-80 score points average higher scores in video game playing.
Key insights
- VLMs with frozen CLIP visual encoders struggle significantly with pixel-level reconstruction (average RE of 20.38), indicating a limitation in capturing fine-grained visual information.
- Adapting the vision encoder during training dramatically improves pixel prediction precision (average RE decreased to 6.65), demonstrating its crucial role in detailed visual perception.
- Integrating pixel reconstruction pre-training with vision encoder adaptation effectively enhances VLMs' capabilities in detail-intensive tasks without compromising general vision-language knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.