Reproduce “Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhening Xing, Gereon Fox, Yanhong Zeng, Xingang Pan, Mohamed Elgharib, Christian Theobalt, Kai Chen
LIVE2DIFF introduces a video diffusion model that employs uni-directional temporal attention and an efficient inference pipeline for live streaming video translation. The method achieves interactive framerates of 16 FPS for 512x512 videos on an RTX 4090 GPU, while maintaining high temporal consistency and structural fidelity to the input.
Key insights
- Uni-directional temporal attention, when combined with a 'warmup area' that provides initial bi-directional context, can effectively enable streaming video generation while maintaining temporal consistency.
- A KV-cache mechanism, similar to those used in LLMs, is critical for achieving interactive framerates in streaming video diffusion by drastically reducing redundant computations of past frames' Key and Value maps.
- Explicitly conditioning the diffusion model with a lightweight depth prior helps preserve the input video's spatial structure, preventing distortions during style or content translation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.