Reproduce “TaleCrafter: Interactive Story Visualization with Multiple Characters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuan Gong, Youxin Pang, Xiaodong Cun, Menghan Xia, Yingqing He, Haoxin Chen, Longyue Wang, Yong Zhang and 3 more
TaleCrafter introduces an interactive system for story visualization that leverages large language models and controllable text-to-image diffusion models to generate consistent visual narratives with multiple characters. It achieves enhanced text-image alignment and identity preservation, while offering interactive control over layout and character poses.
Key insights
- The integration of a large language model (GPT-4) for prompt generation with controllable text-to-image diffusion models allows for robust generalization to novel characters and scenes.
- Personalized LoRA weights and an iterative inpainting process enable efficient and consistent identity preservation for multiple characters within complex scenes.
- Multi-level interactive control over visual elements, including explicit layout guidance via bounding boxes and local structure manipulation via sketches, empowers users with precise creative agency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.