Reproduce “Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization” on OpenResearch

About this paper

Sina Mokhtarzadeh Azar, Emad Bahrami, Enrico Pallotta, Gianpiero Francesca, Radu Timofte, Juergen Gall

SAVi-DNO enables pre-trained diffusion models for video prediction to continuously adapt to live video streams by optimizing diffusion noise during inference. This method improves prediction accuracy and video quality across diverse datasets and diffusion models, offering an efficient and privacy-preserving solution for real-world continuous adaptation.

Key insights

• Continuously optimizing the diffusion noise vector during inference provides an efficient and effective alternative to fine-tuning model parameters for stream adaptation.
• A composite loss, combining pixel-level and high-level feature supervision, is critical for improving both prediction accuracy and semantic coherence in generated video.
• Introducing a controlled amount of randomness into the optimized noise allows for a trade-off between deterministic accuracy and the natural diversity of future video events.

Summary by alphaXiv.