Reproduce “Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Caiming Xiong, Shiva Kumar Pentyala, Chien-Sheng Wu
Salesforce AI Research developed a framework that automatically extracts and evaluates structured dialog workflows from historical customer-agent conversations. This approach uses procedural element-based retrieval and a QA-CoT prompting technique to generate workflows, demonstrating improved accuracy in guiding service AI agents.
Key insights
- Filtering historical conversations based on extracted procedural elements (Proc-Sim) is more effective for identifying high-quality data for workflow extraction than relying on overall conversation similarity or diversity.
- Employing a structured Question-Answer-based Chain-of-Thought (QA-CoT) prompting strategy substantially improves Large Language Models' (LLMs) ability to generate accurate and detailed conditional dialog workflows.
- An End-to-End (E2E) simulation framework offers a reliable and scalable evaluation method for complex branching workflows, demonstrating high agreement with human judgments (Cohen's of 0.922).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.