Reproduce “Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy
Pick-a-Pic introduces a large, open dataset of over half-a-million human preferences for text-to-image generation, derived from real user interactions. Leveraging this data, the research develops PickScore, a CLIP-H-based model that predicts human preferences with 70.5% accuracy, significantly enhancing text-to-image model evaluation and output ranking.
Key insights
- Authentic human feedback, collected from intrinsically motivated users, provides more valuable and diverse preference data than traditional crowd-sourced methods.
- Traditional metrics like FID exhibit a strong negative correlation with human preferences for T2I images, highlighting their inadequacy for evaluating user-centric quality.
- A specialized reward model (PickScore) trained on comprehensive human preference data can achieve 'superhuman' accuracy in predicting user choices and significantly improve generated image quality through ranking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.