Reproduce “FreeRet: MLLMs as Training-Free Retrievers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang and 1 more
FreeRet is a training-free framework that enables off-the-shelf Multimodal Large Language Models (MLLMs) to function as competitive two-stage retrievers. It achieves up to a 13% improvement over strong supervised baselines on multimodal benchmarks like MMEB and MMEB-V2, while preserving the MLLM's original generative and reasoning capabilities.
Key insights
- The final MLP layer in MLLMs, optimized for next-token prediction, can distort semantic features, making the hidden states from earlier layers more suitable for robust retrieval embeddings.
- Prompt engineering alone can effectively steer MLLMs to generate semantically precise and task-aligned embeddings without needing parameter updates or fine-tuning.
- MLLMs exhibit a "framing effect," where the specific lexical format of reranking labels significantly impacts accuracy, a bias that can be mitigated by using a neutral multiple-choice question format.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.