Reproduce “UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lan Chen, Yuchao Gu, Qi Mao
UniVid leverages a pre-trained video generation Diffusion Transformer (DiT) as a universal visual backbone to unify diverse image and video understanding and generation tasks. This framework adapts the model through lightweight fine-tuning with as few as 20 training samples per task, outperforming previous large vision models across various benchmarks including style transfer, depth estimation, and semantic segmentation.
Key insights
- Pre-trained video generation models, initially trained solely for synthesis, possess sufficient inherent visual understanding to serve as a universal backbone for diverse vision tasks.
- The "visual sentence" paradigm effectively unifies image and video tasks, allowing for flexible cross-modal and cross-source generalization based on contextual examples.
- Lightweight adaptation techniques like LoRA enable a pre-trained generative model to achieve state-of-the-art performance on various tasks with very few fine-tuning examples (as few as 20 per task).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.