Reproduce “Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li

The Comprehensive Aesthetic Large language Model (CALM) unifies diverse image aesthetic assessment tasks by integrating multi-scale visual features with a text-guided self-supervised learning approach. CALM achieves state-of-the-art performance across aesthetic scoring, commenting, and personalized assessment, and demonstrates robust zero-shot aesthetic suggestion capabilities, outperforming models like GPT-4v.

Key insights

  • Human aesthetic judgment is multi-faceted, operating across multiple perceptual scales, implying AI systems should integrate information from various levels of visual processing.
  • Different aesthetic attributes (e.g., clarity, composition, semantic content) are encoded at varying depths within a visual encoder, necessitating multi-scale feature extraction.
  • Generating precise attribute-based pseudo-labels through text-guided self-supervision is more effective for aesthetic learning than assuming score degradation from augmentations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.