Reproduce “Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation” on OpenResearch

About this paper

Yuanpei Zhao, Jie Lin, Chao Zhang, Yilin Wang, Mao Li, Chenhui Li, Jie Hou, Tangjie Lv

Researchers introduce PPAINT, a benchmark of Chinese paintings with matched expert pointwise ratings and pairwise preferences, and propose a method to fuse these into a robust ground truth. They also develop PSDISTILL, a label-free self-distillation technique that leverages this fusion principle to train Vision-Language Models for multi-dimensional aesthetic assessment, improving a base VLM's mean SRCC by 40.7% and achieving competitive performance against state-of-the-art models.

Key insights

• Pairwise preferences consistently yield higher inter-expert agreement and ranking stability (e.g., 14.1% higher Kendall's W) for aesthetic judgment compared to pointwise ratings, especially for subjective dimensions.
• Fusing preference-derived ordinal information with high-agreement rating anchors through methods like Anchored Elo or Anchored Davidson Bradley-Terry yields highly convergent and stable absolute aesthetic scores.
• Applying the preference-to-score fusion principle to VLM-generated pseudo-labels effectively creates calibrated training signals, preventing score-scale collapse and significantly improving VLM aesthetic assessment capabilities without expert labels.

Summary by alphaXiv.