Reproduce “Neon: Negative Extrapolation From Self-Training Improves Image Generation” on OpenResearch

About this paper

Sina Alemohammad, Zhangyang Wang, Richard G. Baraniuk

Neon is a post-processing method that enhances generative AI models by inverting the degradation observed during naive self-training. It leverages the "anti-aligned" nature of self-training's degradation gradient to improve performance, demonstrating consistent FID score improvements across diverse architectures with minimal computational overhead and synthetic data.

Key insights

• The degradation observed during naive self-training is not random noise but a structured signal where the synthetic data population gradient is "anti-aligned" with the true data population gradient.
• Common mode-seeking inference samplers (used in autoregressive, diffusion, and flow matching models) introduce a predictable bias that causes this anti-alignment, making negative extrapolation a theoretically sound strategy.
• By inverting this anti-aligned degradation signal, the model can be guided away from over-represented modes and towards a more complete coverage of the true data distribution, reducing real-data risk.

Summary by alphaXiv.