Reproduce “E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Robin Courant, Nicolas Dufour, Xi Wang, Marc Christie, Vicky Kalogeiton
This paper introduces "E.T. the Exceptional Trajectories," a novel approach for generating camera trajectories from textual descriptions while maintaining awareness of character movements. It details the creation of a comprehensive dataset of movie camera and character trajectories and presents "Director," a diffusion-based model that generates high-quality, character-aware, and text-aligned camera movements, outperforming existing methods.
Key insights
- A large, meticulously curated dataset linking cinematographic camera movements to character actions and natural language descriptions is crucial for training generative AI models in this domain.
- Diffusion models, when properly conditioned, can learn the complex distributions of cinematographic camera movements and generate diverse, high-quality trajectories.
- Conditioning generative models on both textual descriptions and character movements enables the creation of character-aware and semantically aligned camera trajectories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.