Reproduce “PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenyi Mo, Tianyu Zhang, Yalong Bai, Ligong Han, Ying Ba, Dimitris N. Metaxas

PREFGEN, a multimodal framework developed by researchers at Rutgers University and iN2X, introduces a method for preference-conditioned image generation by disentangling user aesthetic preferences into distinct representations and integrating them into diffusion models via a novel distribution alignment loss. The method yields superior image quality and preference alignment, evidenced by the highest CLIP Image Score (76.03) and winning over 63% of expert human evaluations against baselines on a new benchmark.

Key insights

  • Multimodal Large Language Models (MLLMs) inherently learn disentangled representations of user preferences, with middle layers encoding stable user identity traits (`e_core`) and top layers capturing context-dependent semantic/stylistic cues (`e_sem`).
  • Aligning embedding distributions using MMD loss is more effective and stable for bridging representational gaps between MLLM preference embeddings and diffusion text encoders compared to restrictive point-wise losses.
  • A comprehensive, unified user representation, concatenating disentangled preference signals (`e_core`, `e_sem`) and a fine-grained visual embedding, allows generative models to maintain prompt fidelity while flexibly adapting to individual aesthetic tastes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.