Reproduce “Python Code Generation by Asking Clarification Questions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haau-Sing Li, Mohsen Mesgar, André F. T. Martins, Iryna Gurevych
An interactive paradigm for text-to-code generation allows systems to resolve under-specified natural language descriptions by asking clarification questions. Providing perfect clarifications leads to substantial improvements in generated code quality, although accurately identifying and ranking these questions presents a challenge.
Key insights
- Clarification questions, when correctly answered, substantially improve the quality and precision of generated code, validating the core hypothesis of the interactive paradigm.
- The synthetic method for generating clarification questions and answers (CQAs) for the CodeClarQA dataset effectively identifies missing key operations and is a scalable approach to data creation.
- The most challenging aspect of building a fully interactive system is accurately predicting the need for clarification and, critically, ranking the most relevant clarification questions, as inaccurate CQs can degrade performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.