Reproduce “DreamRelation: Relation-Centric Video Customization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yujie Wei, Shiwei Zhang, Hangjie Yuan, Biao Gong, Longxiang Tang, Xiang Wang, Haonan Qiu, Hengjia Li and 3 more

DreamRelation, developed by a collaboration including Fudan University and Alibaba Group, introduces a method for generating videos with customized relationships between multiple subjects, such as a dog shaking hands with a cat. This approach disentangles relational dynamics from subject appearances, leading to improved generation of specific interactions as evidenced by higher user study scores for relation alignment.

Key insights

  • The Query and Key features within a Diffusion Transformer's attention mechanism primarily encode structural and relational information, while Value features predominantly capture appearance details.
  • Relationships and subject appearances can be effectively disentangled by using a composite LoRA structure, where different LoRAs are injected into specific parts (Q/K for relations, V for subjects) of the attention mechanism.
  • Prioritizing relational dynamics over static appearances through a space-time contrastive loss helps generate consistent and focused interactions without appearance leakage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.