Reproduce “DiT-Reward: Generative Representations for Text-to-Image Reward Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanming Yang, Guoqing Ma, Bo Wang, Yuan Zhang, Wei Tang, Chenyi Li, Haoyang Huang, Nan Duan

DiT-Reward uses a text-to-image generator’s prompt-conditioned representations to score candidate images, and can score generated latents without decoding them when the policy shares its latent space. It trains a reward head on preference pairs. On HPDv3, it reaches 77.6% pairwise accuracy versus HPSv3’s 76.9% using the same data mixture; this benchmark result does not establish universal human alignment.

Key insights

  • Pretrained generative representations from Diffusion Transformers inherently contain substantial human preference information, transferable to reward modeling even with a frozen backbone.
  • Fine-tuning the entire DiT backbone alongside the reward head significantly boosts performance, demonstrating that preference adaptation enhances the discriminative power of these features.
  • Reward-relevant information is distributed across DiT layers, with middle-to-late layers being most informative, and combining features from different architectural stages provides complementary signals for higher accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.