Reproduce “Character-Centric Understanding of Animated Movies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman
This research from the University of Oxford and Shanghai Jiao Tong University develops a multi-modal pipeline for robust character recognition in animated movies, which leverages automatically constructed audio-visual character banks. The approach improves the accuracy of audio description generation and character-aware subtitling for animated content, addressing challenges posed by diverse character appearances and stylized voices.
Key insights
- Adapting pre-trained vision models like DINOv2 to the animated domain through movie-specific contrastive training significantly enhances visual character identification.
- Jointly leveraging visual and audio cues, particularly using visual context to refine speaker recognition, is effective in overcoming challenges like stylized character voices and poor audio quality.
- Automatically building extensive multi-modal character banks, including in-movie speech segments, provides crucial complementary information for robust and scalable character understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.