Reproduce “Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shizhan Liu, Xinran Deng, Zhuoyi Yang, Jiayan Teng, Xiaotao Gu, Jie Tang
Spectral-Structured VAE (SSVAE) improves the usefulness of video latents for downstream diffusion by shaping both their channel covariance and spatio-temporal frequency content. Through Latent Masked Reconstruction and Local Correlation Regularization, the method reports faster convergence and higher text-to-video quality, including a roughly threefold convergence speedup and UnifiedReward gains from 40.9 to 45.9 in a 17-frame, 512×512 setting, while accepting a small reduction in reconstruction fidelity.
Key insights
- The channel eigenspectrum and spatio-temporal power spectrum are proposed as direct indicators of latent diffusability: useful video latents need not be spectrally uniform, but should exhibit controlled concentration in dominant modes and low frequencies.
- For flow matching, the derived input-target cross-correlation shares eigenvectors with the clean-latent covariance, linking latent spectral concentration to the relative learning strength of diffusion modes.
- Generation quality can improve even when reconstruction metrics decline slightly, indicating that a reconstruction-optimal tokenizer is not necessarily the best tokenizer for latent video generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.