Reproduce “Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhang Ma, Wenting Xu, Chaoyi Zhao, Keqiang Sun, Qinfeng Jin, Zeng Zhao, Changjie Fan, Zhipeng Hu
Storynizor introduces a framework for consistent story generation, allowing multiple characters to maintain identity and appearance across a sequence of images while exhibiting diverse poses and distinct backgrounds. It achieves this by employing an ID-Synchronizer for inter-frame coherence and an ID-Injector for robust identity preservation, trained on the newly curated StoryDB dataset.
Key insights
- Leveraging attention masks in a spatial self-attention mechanism (AMSA) effectively enforces character consistency across frames while allowing backgrounds to vary.
- Applying a perceptual loss to cross-attention maps (MPL) refines character localization and enhances the disentanglement of foreground and background elements.
- Randomly shuffling reference images of the same character ID during training (SRS) compels the model to learn pose-invariant identity features, leading to greater pose diversity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.