Reproduce “ARAML: A Stable Adversarial Training Framework for Text Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pei Ke, Fei Huang, Minlie Huang, Xiaoyan Zhu

The ARAML framework from Tsinghua University provides a stable adversarial training method for text generation by combining a reward-augmented maximum likelihood objective with sampling from a stationary distribution. This approach yields generated text with improved quality and diversity, demonstrating reduced training instability across various benchmarks.

Key insights

  • Connecting the reinforcement learning objective to a KL divergence allows for a stable, MLE-like training approach for the generator.
  • Sampling from a stationary distribution, constructed by perturbing real data and optionally guided by a language model, offers a stable training signal and prevents the generator from diverging.
  • Replacing fixed rewards with a learnable discriminator's output within the RAML framework effectively addresses exposure bias while maintaining training stability for discrete text.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.