Reproduce “SPICE: Semantic Propositional Image Caption Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
SPICE is an image caption evaluation metric that assesses semantic content by converting captions into scene graphs rather than relying on word overlap. It achieves significantly higher correlation with human judgments than n-gram based metrics and provides detailed diagnostic insights into model capabilities, such as color description or counting accuracy.
Key insights
- Semantic propositional content is a crucial component of human caption evaluation, which existing n-gram based metrics largely miss.
- By decomposing evaluation into specific semantic categories, SPICE provides diagnostic power, revealing strengths and weaknesses of captioning models (e.g., color perception vs. counting).
- The metric's design is not tied to a specific parsing pipeline, allowing for future improvements as syntactic and semantic parsing technologies advance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.