Reproduce “Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation” on OpenResearch

About this paper

Ruizhe Li, Xiao Li, Guanyi Chen, Chenghua Lin

The TWR-VAE framework addresses posterior collapse in RNN-based Variational Autoencoders for text modeling by applying KL regularization to latent variables from every timestep. This approach consistently enhanced performance, reducing negative log-likelihood by 11.8 and perplexity by 24.1 on average over strong baselines in language modeling, while also improving diversity and relevance in dialogue response generation.

Key insights

• Regularizing latent variables across all timesteps of an RNN encoder, rather than only the last, is crucial for mitigating posterior collapse and encouraging the learning of robust data representations.
• Learning robust data representations, particularly during the early stages of decoder training, is essential for enabling the decoder to generate diverse and controllable observations.
• The proposed timestep-wise regularization offers a simple, generic architectural modification that seamlessly integrates with existing RNN-based VAEs without requiring complex tuning or additional components.

Summary by alphaXiv.