Reproduce “VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, Feng Yang
Google Research developed VILA, a two-stage framework that leverages vision-language pretraining to learn image aesthetics directly from natural language user comments, rather than solely relying on numerical scores. The system achieves state-of-the-art performance on the AVA dataset for Image Aesthetic Assessment with an SRCC of 0.774 and demonstrates robust zero-shot generalization for aesthetic evaluation and style classification.
Key insights
- Natural language user comments provide a significantly richer and more nuanced source of aesthetic supervision than simple numerical scores, allowing models to learn the 'why' behind aesthetic judgments.
- Targeted aesthetic-specific vision-language pretraining is crucial for extracting and embedding comprehensive aesthetic semantics, as general web-crawled data dilutes aesthetic information.
- A lightweight, rank-based adapter combined with a frozen VLM backbone can efficiently achieve state-of-the-art performance on score-based IAA tasks while maintaining strong zero-shot generalization capabilities across various aesthetic concepts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.