Reproduce “StoryGAN: A Sequential Conditional GAN for Story Visualization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson and 1 more
StoryGAN, a sequential conditional Generative Adversarial Network, addresses the task of visualizing multi-sentence narratives by generating coherent image sequences. It achieves both local image quality and global story consistency, demonstrating superior performance with a 0.672 SSIM score on CLEVR-SV and receiving overwhelming preference in human evaluations on Pororo-SV over baseline methods.
Key insights
- Effectively visualizing multi-sentence stories requires a model to maintain global narrative consistency across discrete, potentially scene-changing frames, a challenge distinct from single-image or continuous video generation.
- Dynamically integrating both local (current sentence) and global (accumulated story) context via a recurrent mechanism is critical for generating coherent sequential images that evolve logically with the narrative.
- Employing a multi-tiered adversarial approach, with discriminators at both the individual image and full-story sequence levels, significantly enhances the training process for achieving both local fidelity and global narrative coherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.