Reproduce “Autoencoding Under Normalization Constraints” on OpenResearch

About this paper

Sangwoong Yoon, Yung-Kyun Noh, Frank Chongwoo Park

This paper presents the Normalized Autoencoder (NAE), a probabilistic model that reinterprets an autoencoder's reconstruction error as an energy function to impose normalization constraints. NAE, trained with a novel On-Manifold Initialization (OMI) sampling strategy, addresses the 'outlier reconstruction' problem in conventional autoencoders and demonstrates superior performance in outlier detection, achieving an average AUC of 0.955 on MNIST hold-out classes.

Key insights

• The core issue of conventional autoencoders in outlier detection is their lack of a probabilistic normalization constraint, which allows them to assign low reconstruction error to out-of-distribution samples.
• Reframing an autoencoder's reconstruction error as an energy function within an Energy-Based Model framework allows for the enforcement of proper probability normalization, actively suppressing outlier reconstruction.
• The On-Manifold Initialization (OMI) sampling strategy effectively addresses the convergence and exploration issues of MCMC training for EBMs by leveraging the autoencoder's decoder manifold for intelligent initialization.

Summary by alphaXiv.