Reproduce “ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context” on OpenResearch

About this paper

Sixiao Zheng, Yanwei Fu

A non-autoregressive framework for visual storytelling, ContextualStory, generates coherent story frames from textual storylines by integrating spatially-enhanced temporal attention, a storyline contextualizer, and a storyflow adapter into a latent diffusion model. Developed at Fudan University, the framework improves character and scene consistency, achieves higher image quality (e.g., lower FID of 13.61 on PororoSV), and significantly reduces inference time (11.8 seconds) compared to state-of-the-art autoregressive methods.

Key insights

• A non-autoregressive framework, when designed with comprehensive contextual awareness, can effectively overcome the memory, speed, and consistency limitations inherent in autoregressive visual storytelling models.
• Explicitly capturing both localized spatial dependencies within frames and temporal dependencies of moving characters across frames, as achieved by SETA, is critical for maintaining visual consistency in dynamic narratives.
• Enriching individual sentence embeddings with a global understanding of the entire storyline significantly improves narrative coherence and the overall visual consistency of generated sequences.
• Guiding the generation process based on quantified scene changes between frames (StoryFlow Adapter) can lead to more natural and contextually appropriate visual transitions.

Summary by alphaXiv.