Reproduce “CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation” on OpenResearch

About this paper

Reza Abbasi, Ali Nazari, Aminreza Sefid, Mohammadali Banayeeanzade, Mohammad Hossein Rohban, Mahdieh Soleymani Baghshah

Researchers from Sharif University of Technology performed a fine-grained analysis of CLIP's multi-object representation, revealing consistent biases where the text encoder prioritizes first-mentioned objects and the image encoder emphasizes larger objects. These biases lead to reduced performance in image-text matching for semantically equivalent but structurally varied inputs, and affect object prominence in text-to-image generation.

Key insights

• CLIP's text encoder consistently assigns disproportionately high importance to the first object mentioned in multi-object captions.
• The image encoder exhibits a strong bias towards visually larger objects within multi-object images, a characteristic linked to the Vision Transformer architecture.
• The text encoder's bias is not solely from its attention mechanism but arises from CLIP's contrastive training, which transfers image-side bias, and is reinforced by object-size-to-mention-order biases prevalent in training datasets like LAION.

Summary by alphaXiv.