Reproduce “NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi
NEUROLOGIC DECODING is an inference-time algorithm that enables neural language models to generate fluent text while consistently satisfying complex lexical constraints expressed in predicate logic. This method achieves an efficient O(N k) runtime and demonstrates improved constraint satisfaction and generation quality across various tasks, with unsupervised models often outperforming larger supervised models using conventional decoding.
Key insights
- Inference-time algorithms can provide a robust, general, and computationally efficient avenue for highly controllable text generation, complementing or even surpassing the capabilities of extensive supervised finetuning for constraint satisfaction.
- The challenge of reliably enforcing complex lexical constraints in LMs is often an 'under-specification' issue in finetuning, where models learn superficial patterns but not explicit logical rules.
- Tracking the dynamic satisfaction status of complex logical constraints and selectively rewarding progress during beam search allows for efficient enforcement without degrading output fluency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.