Reproduce “Optimization Benchmark for Diffusion Models on Dynamical Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabian Schaipp
This research introduces the first dedicated optimization benchmark for diffusion models, demonstrating that modern optimizers like Muon and SOAP achieve approximately 18% lower final validation loss compared to AdamW, and revealing that generative quality does not always correlate directly with validation loss in diffusion model training.
Key insights
- Advanced optimizers like Muon and SOAP demonstrate superior performance in diffusion model training, achieving substantially lower final losses compared to AdamW.
- The overall training trajectory and choice of learning-rate schedule significantly influence generative quality, with some schedules optimizing loss but yielding less stable or lower visual fidelity.
- The performance gap between Adam-type optimizers and SGD persists in diffusion models even when class imbalance is not a factor, suggesting other underlying causes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.