Reproduce “Image Fusion via Vision-Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zixiang Zhao, Lilun Deng, Haowen Bai, Yukun Cui, Zhipeng Zhang, Yulun Zhang, Haotong Qin, Dongdong Chen and 3 more
Introducing Image Fusion via Vision-Language Model (FILM), a novel framework leverages explicit textual descriptions from large language models to semantically guide the image fusion process. FILM consistently achieves superior performance over state-of-the-art methods across four diverse image fusion tasks, including infrared-visible and medical imaging, and contributes a new Vision-Language Fusion (VLF) dataset.
Key insights
- Explicit textual guidance from large language models provides a deeper semantic understanding that significantly enhances image fusion beyond purely visual cues.
- Leveraging cross-modal attention, where textual information guides visual feature extraction, enables more informed and contextually aware fusion decisions.
- Synthesizing multi-grained visual prompts (captions, dense captions, masks) via an LLM like ChatGPT generates rich, coherent semantic narratives crucial for effective guidance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.