Reproduce “Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Junshi Huang
This paper introduces Diffusion-RWKV, an architecture that leverages RWKV models for diffusion-based image generation to address the quadratic computational complexity of Transformer-based models. The proposed model achieves competitive FID scores on standard benchmarks like CIFAR-10 and ImageNet while demonstrating linear complexity, reducing FLOPs by 30-40% compared to Transformer models for 32x32 images.
Key insights
- Smaller patch sizes (e.g., 2x2) generally yield better image generation quality but increase computational cost due to longer sequence lengths.
- Concatenation-based skip connections between shallow and deep layers improve information flow and lead to superior performance compared to addition-based or no skip connections.
- adaLN-Zero conditioning consistently outperforms other conditioning methods like adaLN and in-context conditioning, particularly as training progresses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.