Reproduce “RPiAE: A Representation-Pivoted Autoencoder Enhancing Both Image Generation and Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yue Gong, Hongyu Li, Shanyuan Liu, Bo Cheng, Yuhang Ma, Liebucha Wu, Xiaoyu Wu, Manyuan Zhang and 3 more

RPiAE is a representation-pivoted autoencoder that enhances both image generation and editing within latent diffusion models by enabling reconstruction-oriented fine-tuning of a representation-initialized encoder while preserving its semantic structure. The system achieved state-of-the-art performance in class-conditional generation with a gFID of 1.09 on ImageNet-1K, and outperformed baselines in text-to-image synthesis and image editing benchmarks.

Key insights

  • It is possible to fine-tune a representation-initialized encoder for improved reconstruction fidelity without corrupting its inherited semantic structure, by employing a pivot regularization mechanism.
  • Decoupling the training objectives into sequential stages allows for effective optimization of competing requirements (semantic preservation, reconstruction, generative tractability) in the autoencoder's latent space.
  • Compressing high-dimensional representation features into a compact, lower-dimensional and KL-regularized latent space improves generative tractability for downstream diffusion models while retaining critical semantic information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.