Reproduce “ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingyang Wu, Ashirbad Mishra, Soumik Dey, Shuo Xing, Naveen Ravipati, Hansi Wu, Binbin Li, Zhengzhong Tu

Researchers from Texas A&M University and eBay Inc. introduced ConsID-Gen, an Image-to-Video framework that excels at preserving fine-grained object identity and geometric consistency across varying viewpoints. This work also includes ConsIDVid, a large-scale object-centric video dataset, and ConsIDVid-Bench, a new benchmark for evaluating identity preservation, with the framework outperforming leading I2V models and achieving identity consistency comparable to proprietary systems.

Key insights

  • Explicitly leveraging unposed multi-view reference images significantly enriches structural context, which is crucial for anchoring object shape and appearance during video generation.
  • A dual-path visual representation, combining a semantic 2D encoder with a geometry-aware encoder and integrated through a multi-modal module, is effective in mitigating appearance drift and geometric distortion.
  • Early and sophisticated fusion of visual and textual features is essential for robust identity preservation, as demonstrated by the multi-modal interaction modules that refine cross-modal alignment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.