Reproduce “Microsoft COCO Captions: Data Collection and Evaluation Server” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, C. Lawrence Zitnick
The Microsoft COCO Captions dataset, featuring over 1 million human-generated captions for 160,000 diverse images, and its associated standardized evaluation server were introduced to benchmark automatic image captioning algorithms. This work quantified human agreement in image description, reporting a CIDEr-D score of 0.854 for human performance on the c5 subset.
Key insights
- Architectural improvements in neural networks and the availability of large datasets were key drivers for renewed interest and progress in image captioning.
- Human annotators demonstrate quantifiable consistency in describing images, which can be modeled by a simple probabilistic approach relating word usage to object presence.
- The number of reference captions significantly impacts automatic evaluation scores, with more references (e.g., c40) leading to higher human agreement scores, underscoring the need for rich evaluation data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.