Reproduce “Visual Storytelling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ting-Hao, Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick and 8 more
This research introduces the task of visual storytelling by presenting SIND v.1, a novel dataset of 20,211 image sequences with tiered language annotations for literal descriptions and cohesive narratives. Baseline sequence-to-sequence RNN models, combined with decode-time heuristics, demonstrated improved narrative generation quality, achieving a METEOR score of 31.42 through strategies like greedy search and deduplication.
Key insights
- Narrative language (SIS) exhibits distinct lexical characteristics, incorporating more temporal, abstract, and inferential elements compared to single-image (DII) or in-sequence (DIS) descriptions.
- METEOR was identified as the most suitable automatic evaluation metric for visual storytelling, showing stronger correlation with human judgments than BLEU or Skip-Thoughts.
- Optimal decoding strategies for story generation differ from image captioning, with greedy search and specific heuristics (deduplication, visually grounded words) significantly improving narrative quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.