Reproduce “DeRA: Decoupled Representation Alignment for Video Tokenization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengbo Guo, Junke Wang, Zhen Xing, Chengxu Liu, Daoguo Dong, Xueming Qian, Zuxuan Wu

Efficiently creating compact and semantically rich video representations for generative models, DeRA introduces a 1D video tokenizer that explicitly disentangles spatial appearance and temporal motion representations using distinct vision foundation models as supervisors. This approach enhances training efficiency and achieves state-of-the-art performance in video reconstruction, reducing rFVD on UCF-101 by 25% compared to prior methods, and setting new benchmarks for autoregressive video generation.

Key insights

  • Explicitly decoupling spatial and temporal representation learning, guided by specialized foundation models, significantly improves the efficiency and performance of video tokenization.
  • Knowledge distillation from distinct pre-trained image and video foundation models provides effective, complementary supervisory signals for learning disentangled latent spaces.
  • Addressing gradient conflicts through a novel projection mechanism (SACP) is crucial for stable and optimal training when multiple, potentially conflicting, alignment objectives are present.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.