Reproduce “Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin
A cyclical annealing schedule for the KL divergence term effectively mitigates the KL vanishing problem in Variational Autoencoders (VAEs) for natural language processing, fostering the learning of more informative latent representations and improving performance in tasks such as language modeling and dialogue generation.
Key insights
- Autoregressive VAE decoders operate via two paths: a latent-conditioned path (`p(x|z)`) and an autoregressive path (`p(x_t|x<t, z)`), with KL vanishing occurring when the decoder prefers the "easier" autoregressive path by ignoring an uninformative 'z'.
- Each subsequent cycle of the annealing schedule uses the progressively improved latent codes from the previous cycle as a warm restart, thereby continuously refining the utility and structure of the latent space.
- The analytical decomposition of the KL divergence term shows that maximizing mutual information between 'z' and 'x' (reducing KL vanishing) is inversely proportional to 'β', justifying the strategic manipulation of 'β' to first encourage information capture and then regularization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.