Reproduce “Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Up” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lang Huang, Qiyu Wu, Zhongtao Miao, Toshihiko Yamasaki

This paper introduces the Joint Fusion Encoder (JFE), a unified one-tower architecture for multimodal retrieval that integrates visual and textual information early within a single MLLM backbone. The approach outperforms traditional late-fusion two-tower models, particularly for complex mixed-modal, multi-modal, and conditional retrieval tasks, establishing a new state of the art on benchmarks like M-BEIR and GeneCIS.

Key insights

  • Early and joint fusion of visual and textual information within a single model architecture significantly enhances understanding for complex multimodal retrieval tasks.
  • Adapting pre-trained Multimodal Large Language Models (MLLMs) through a two-stage process (post-training adaptation and instruction tuning) effectively transforms them into powerful discriminative encoders for retrieval.
  • A carefully designed data sampling strategy and optimal LoRA hyperparameters are crucial for effectively fine-tuning MLLMs for multi-task multimodal retrieval.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.