Reproduce “ToonCrafter: Generative Cartoon Interpolation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong
ToonCrafter presents a generative approach for cartoon video frame interpolation, leveraging diffusion models to overcome the limitations of traditional methods. It achieves superior visual quality and temporal consistency by adapting live-action priors to cartoon aesthetics, preserving fine details through a novel decoder, and offering sketch-based user control, streamlining the animation production process.
Key insights
- Generative models, specifically diffusion models, can overcome the limitations of traditional correspondence-based methods for stylized content like cartoon animation by synthesizing new content and handling non-linear motions.
- Selective fine-tuning strategies can effectively adapt large pre-trained live-action diffusion models to new domains (cartoons) while preserving robust motion priors and preventing catastrophic forgetting.
- Incorporating detail injection and propagation mechanisms within a dual-reference decoder is essential to recover and maintain fine structural details in high-contrast content like cartoons when operating in compressed latent spaces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.