Reproduce “DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song and 4 more
A framework called DEEM employs diffusion models to provide self-supervised generative feedback, enhancing the image perception capabilities of Large Multimodal Models (LMMs). This approach bolsters LMM robustness against out-of-distribution data and mitigates visual hallucinations, often outperforming larger state-of-the-art models with fewer resources.
Key insights
- Diffusion models can effectively serve as a generative feedback mechanism to refine the perceptual capabilities of LMM image encoders, moving beyond their role solely in image generation.
- Correcting semantic biases and enhancing detail retention in image encoders through self-supervised generative checks significantly improves LMM robustness and reduces visual hallucinations.
- Resource-efficient approaches, by intelligently leveraging existing model components, can achieve superior perceptual performance compared to simply scaling up model size or training data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.