Reproduce “Multi-modal Learnable Queries for Image Aesthetics Assessment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiwei Xiong, Yunfan Zhang, Zhiqi Shen, Peiran Ren, Han Yu
Researchers from Nanyang Technological University and Alibaba Group developed MMLQ, a multi-modal framework that leverages learnable queries on frozen pre-trained image and text encoders to assess image aesthetics. The framework achieves new state-of-the-art performance on the AVA/AVA-Comments dataset, improving SRCC by 7.7% and PLCC by 8.3% over previous multi-modal methods.
Key insights
- Learnable queries effectively extract diverse aesthetic patterns from multi-modal inputs (images and comments) using frozen, general-purpose pre-trained encoders without needing explicit aesthetic attribute labels.
- Textual context from user comments provides highly discriminative aesthetic guidance, surprisingly outperforming visual-only assessment in some cases, highlighting the value of human feedback.
- Optimal performance is achieved with a minimal number of learnable queries and stable performance across varying numbers of MMIBs, suggesting an efficient and focused architecture for aesthetic feature learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.