Reproduce “Plan with Code: Comparing approaches for robust NL to DSL generation” on OpenResearch

About this paper

Nastaran Bassamzadeh, Chhaya Methani

Researchers at Microsoft Corporation introduced an optimized Retrieval Augmented Generation (RAG) framework for converting natural language to Domain Specific Languages (NL2DSL) for workflow automation. This RAG approach, combining dynamic few-shot selection with API metadata, demonstrated superior performance for out-of-domain APIs, increasing average similarity by 7 percentage points and reducing API hallucinations by nearly 2 percentage points compared to fine-tuned models.

Key insights

• Dynamically selected few-shot examples combined with explicit API function definitions are crucial for enhancing both syntactic correctness and reducing hallucinations in RAG-based DSL generation.
• The RAG approach effectively addresses the challenge of out-of-domain APIs, offering a robust and adaptable solution for evolving enterprise API ecosystems without constant fine-tuning.
• While fine-tuned models can perform well on in-domain data, RAG strategies provide a comparable or superior solution for real-world scenarios with unseen APIs, minimizing operational overhead.

Summary by alphaXiv.