Reproduce “Constantly Improving Image Models Need Constantly Improving Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxin Ge, Grace Luo, Heekyung Lee, Nishant Malpani, Long Lian, XuDong Wang, Aleksander Holynski, Trevor Darrell and 2 more
The ECHO framework systematically converts qualitative user interactions and feedback from social media into dynamic, quantifiable benchmarks for image generation models. This approach reveals novel and complex user tasks, effectively differentiates state-of-the-art models, and provides actionable insights for model development based on real-world usage.
Key insights
- Real-world user interactions on social media reveal diverse, complex, and emergent tasks (e.g., Ghiblification) that static benchmarks fail to capture, highlighting a disconnect in model evaluation.
- Qualitative community feedback contains rich, actionable insights into model capabilities, common failure modes (e.g., identity shift, color drift), and even user-developed strategies to overcome limitations.
- Leveraging social media data allows for the creation of benchmarks with more natural language prompts and greater linguistic diversity, better reflecting how users actually interact with generative models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.