Reproduce “SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Drama” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing Tang, Quanlu Jia, Yuqiang Xie, Zeyu Gong, Xiang Wen, Jiayi Zhang, Yalong Guo, Guibin Chen and 1 more
SkyScript-100M introduces a colossal dataset of 1,000,000,000 paired scripts and detailed shooting scripts for short dramas, derived from 10 million meticulously annotated shooting scripts, to advance AI-powered video generation. The dataset's rich structure, including 3D object locations, continuous emotion scores, and camera parameters, enabled a custom model, SkyReels, to generate videos with improved character consistency and plot understanding compared to other advanced LLMs.
Key insights
- Feeding video generation models with rich "world consensus" information, including detailed 3D layouts, continuous emotion (VAD scores), and highlight scores, is crucial for maintaining character consistency and narrative coherence.
- Integrating open-vocabulary detection and specialized character models helps ground MLLM output visually, addressing hallucinations and providing accurate spatial and emotional context.
- Implicit character relationships and continuous video highlights can be mined from richly annotated script-shooting script pairs, enabling more complex plot generation and nuanced emotional storytelling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.