Reproduce “Autoregressive Text Generation Beyond Feedback Loops” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Florian Schmidt, Stephan Mandt, Thomas Hofmann
Researchers from ETH Zürich and UC Irvine developed a latent state space model with a Conditional Random Field (CRF) observation model for text generation, addressing exposure bias by making latent state evolution non-autoregressive with respect to past observations. This model effectively captures local word correlations, leading to generated sentences with improved fluency and significantly reduced repetitions compared to autoregressive baselines, while maintaining high diversity.
Key insights
- Decoupling latent state evolution from direct observation feedback effectively mitigates exposure bias at the state level, leading to more robust generative models.
- Globally normalized Conditional Random Fields can effectively capture local word dependencies and improve text coherence without relying on autoregressive feedback into the latent state.
- An embedding-based factorization technique significantly enhances the computational tractability and efficiency of CRFs for high-dimensional output spaces, making them practical for text generation with large vocabularies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.