Reproduce “SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaofu Chen, Israfel Salazar, Yova Kementchedjhieva
SPECS, a reference-free Representational Similarity metric, evaluates long image captions with high human correlation by enhancing CLIPScore's sensitivity to fine-grained details. It offers a computationally efficient alternative to expensive LLM-based metrics, making it suitable for iterative model development in vision-language tasks.
Key insights
- Explicit training for "specificity," defined by the accurate inclusion and exclusion of fine-grained details, drastically improves human correlation for RS metrics on long captions.
- Enhancing specificity can introduce hubness into the embedding space, leading to a trade-off with performance on general vision-language tasks like retrieval and classification.
- Evaluation metrics show sensitivity to caption length, with SPECS performing optimally for captions over 60 tokens, while CLIPScore is better suited for shorter descriptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.