Reproduce “SEED-Story: Multimodal Long Story Generation with Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, Yingcong Chen
Researchers from HKUST, CUHK, and Tencent labs developed SEED-STORY, a framework for generating long multimodal stories by combining a Multimodal Large Language Model (MLLM) with an efficient multimodal attention sink mechanism. The system produces coherent narratives with visually consistent images for extended sequences, and is supported by the new large-scale StoryStream dataset.
Key insights
- MLLMs can be effectively extended for complex, multi-turn, and long-form multimodal creative generation tasks by integrating robust visual generation components and specialized training pipelines.
- Careful and selective retention of key tokens (e.g., start tokens, beginning/end-of-image tokens) in the KV cache is crucial for maintaining long-term coherence and preventing quality degradation in extended MLLM generations.
- High-quality, domain-specific datasets with rich narratives, high-resolution images, and inherent character/style consistency are essential for advancing multimodal story generation capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.