Reproduce “Diffusion Transformers with Representation Autoencoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie
The paper introduces Representation Autoencoders for image diffusion, replacing the compressed VAE latent with tokens from a frozen vision encoder and a trained decoder. It adapts the denoiser through width matching, noise-schedule shifts, noisy-latent decoder training, and a wide head. On class-conditional ImageNet, it reaches gFID 1.51 unguided and 1.13 guided at 256×256, but evidence is limited to the setting.
Key insights
- Semantic encoders, when paired with an efficiently trained decoder, can achieve superior pixel-level reconstruction and provide semantically rich latent spaces, disproving prior assumptions.
- Effective training of Diffusion Transformers with high-dimensional latents requires matching DiT model width to the latent token dimensionality and adjusting noise schedules based on the effective data dimension.
- Noise-augmented decoding significantly improves generative fidelity by making the RAE decoder robust to the continuous and noisy outputs inherent to diffusion model inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.