Reproduce “MMPB: It's Time for Multi-Modal Personalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jaeik Kim, Woojin Kim, Woohyeon Park, Jaeyoung Do

This research introduces MMPB, a comprehensive benchmark designed to evaluate how Vision-Language Models (VLMs) adapt to individual users' specific visual concepts and preferences. The evaluation of 23 VLMs on MMPB reveals that current models struggle with preference-grounded reasoning, exhibit safety-induced evasiveness, and fail to effectively leverage visual cues or maintain personalized context over extended interactions.

Key insights

  • Current VLMs demonstrate a fundamental limitation in preference-grounded reasoning, struggling with abductive inference required to integrate user intentions and tastes with visual understanding.
  • Many closed-source and some open-source VLMs frequently exhibit 'safety-induced evasiveness' through refusals on human-centric personalization queries, hindering practical personalization utility.
  • VLMs do not effectively leverage multiple fine-grained visual examples for personalization, with simple textual concept injections often achieving comparable or superior performance to image-based methods.
  • Severe long-context forgetting and positional bias were observed, as VLMs struggled to retain personalized concepts over extended multi-turn interactions or when presented with multiple concepts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.