Reproduce “InstanceV: Instance-Level Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuheng Chen, Teng Hu, Jiangning Zhang, Zhucun Xue, Ran Yi, Lizhuang Ma
InstanceV, developed by researchers from Shanghai Jiao Tong University and Zhejiang University, introduces instance-level control to text-to-video generation by modifying the Diffusion Transformer architecture with instance-aware attention, prompt enhancement, and guidance modules. This framework achieves enhanced video fidelity and consistency, reflected by a Fréchet Video Distance of 2313.06 and an Area-Weighted Mean IoU of 0.7298 on the InstanceBench benchmark, enabling precise control over object attributes and spatial layouts.
Key insights
- Integrating instance-level control directly into the Diffusion Transformer architecture through masked cross-attention allows for precise spatial and attribute control without complex feature fusion or post-processing.
- Enhancing individual instance prompts with global video context via a shared, timestep-adaptive module is crucial for maintaining overall semantic consistency and preventing generation artifacts.
- Applying spatial awareness during unconditional guidance, even without semantic content, helps prevent small objects from being overlooked and improves their fidelity in complex scenes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.