Reproduce “RefAlign: Representation Alignment for Reference-to-Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lei Wang, YuXin Song, Ge Wu, Haocheng Feng, Hang Zhou, Jingdong Wang, Yaxing Wang, Jian Yang

RefAlign introduces a representation alignment framework for reference-to-video (R2V) generation, which explicitly aligns internal Diffusion Transformer (DiT) reference features to a frozen Vision Foundation Model (VFM) during training. This method enhances subject identity consistency and semantic discriminability, achieving state-of-the-art results on the OpenS2V-Eval benchmark without adding inference overhead.

Key insights

  • Explicitly aligning DiT's internal reference representations to a robust VFM semantic space effectively mitigates 'modality mismatch' and improves subject identity preservation and semantic separation.
  • The combination of positive and negative alignment terms in the RA loss is critical; the negative term prevents representation collapse and multi-subject confusion by enforcing inter-subject separability.
  • Vision Foundation Models like DINOv3 provide a strong and spatially consistent semantic anchor for alignment, proving more effective than other VFMs or simply concatenating VFM features as input due to better modality matching and stable fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.