Reproduce “Synthesis of Mathematical programs from Natural Language Specifications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ganesh Prasath, Shirish Karande

Researchers from Tata Consultancy Services Research developed an automated system to synthesize mathematical programs from natural language, bridging natural language processing and operations research. Their enhanced CodeT5-base model achieved an execution accuracy of 0.73, outperforming ChatGPT (0.41) and Codex (0.36), and a canonical accuracy of 89.63% (pass@5), exceeding prior state-of-the-art.

Key insights

  • Transformer-based language models, specifically CodeT5, can be effectively fine-tuned and enhanced with domain knowledge for high-accuracy mathematical program generation.
  • Data augmentation via reverse translation with large language models, even with noisy outputs, can substantially improve performance in data-scarce specialized domains.
  • A hybrid approach combining neural generation with symbolic reasoning through custom beam scoring and rule-based post-processing is effective for enforcing domain-specific correctness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.