Reproduce “PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models” on OpenResearch

About this paper

Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau

PICARD (Parsing Incrementally for Constrained Auto-Regressive Decoding) enables language models to generate syntactically and semantically valid formal language outputs like SQL by incrementally parsing and pruning invalid tokens during inference. This method achieves state-of-the-art performance on text-to-SQL tasks, such as Spider and CoSQL, while remaining compatible with generic pre-trained models.

Key insights

• Enforcing formal language constraints incrementally during auto-regressive decoding is a more efficient and effective approach than architectural modifications or post-generation filtering.
• The 'Parsing with Guards' mode, which includes comprehensive semantic checks during Abstract Syntax Tree (AST) assembly, provides the highest performance by 'failing fast' and pruning invalid hypotheses early.
• PICARD's model-agnostic nature allows it to be applied to any auto-regressive language model and its standard vocabulary, including large pre-trained Transformer models, without requiring custom designs.

Summary by alphaXiv.