Reproduce “Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youngjun Jun, Seil Kang, Woojung Han, Seong Jae Hwang
Researchers from Yonsei University developed Interpretable Motion-Attentive Maps (IMAP) to spatially and temporally localize motion concepts within Video Diffusion Transformers. This training-free and gradient-free method achieved an average Motion Localization Score of 0.62 on CogVideoX-5B, outperforming existing baselines and providing insights into how these models generate dynamic movements.
Key insights
- Specific attention heads within Video DiTs exhibit distinct temporal variation in processed features, allowing for their selection to isolate motion-related information.
- Later timesteps in the diffusion process and layers with higher second-largest eigenvalues of the attention weight matrix are crucial for extracting interpretable spatio-temporal semantic features.
- The GramCol approach provides robust and positively highlighted similarity maps for concepts, proving more suitable for frame-varying patterns compared to methods relying on softmax over multiple concepts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.