Reproduce “RFMI: Estimating Mutual Information on Rectified Flow for Text-to-Image Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chao Wang, Giulio Franzese, Alessandro Finamore, Pietro Michiardi

Researchers from EURECOM and Huawei Technologies developed RFMI, a mutual information estimator for Rectified Flow (RF) models, which enables a self-supervised fine-tuning approach (RFMI FT) to improve text-to-image alignment. RFMI FT demonstrates improved performance across various alignment benchmarks like T2I-CompBench++, particularly for 3D-spatial and numeracy challenges, while maintaining image quality.

Key insights

  • The internal structure and velocity field of Rectified Flow (RF) models can be leveraged to directly estimate mutual information (MI) between generated images and text prompts.
  • A self-supervised fine-tuning approach can effectively improve text-to-image alignment by selecting high-MI samples generated by the model itself for training, eliminating the need for external data.
  • Parameter-efficient fine-tuning techniques like LoRA can enhance specific aspects like alignment while preserving the overall generative capabilities and image quality of pre-trained RF models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.