Reproduce “StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengguang Zhou, Jing Li, Huaxia Li, Nemo Chen, Xu Tang
A tuning-free text-to-image generation method, StoryMaker, achieves holistic character consistency, maintaining consistent faces, clothing, hairstyles, and body attributes across multiple generated images. Developed by Xiaohongshu Inc., the model facilitates coherent visual storytelling by producing sequential images with unified characters, evidenced by a CLIP Image Similarity score of 79.51% and robust multi-character generation.
Key insights
- Achieving holistic character consistency requires dedicated embeddings that capture not just facial features but also clothing, hairstyles, and body attributes.
- Explicitly disentangling individual characters and the background through a masked attention loss is critical for preventing identity intermingling in multi-character scenes.
- Decoupling character pose from reference images using techniques like ControlNet allows for greater creative flexibility and diverse pose generation while maintaining character identity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.