Reproduce “CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban and 1 more
CRMArena tests whether AI agents can complete customer-service work in a populated Salesforce environment. It uses nine tasks across 16 linked record types, with scenarios informed by Salesforce users. The best overall averages were 57.7% for o1 using ReAct and 64.3% for o1 using function calling, showing incomplete performance on this synthetic shoe-company benchmark.
Key insights
- The CRMArena environment, including its complex data model and tasks, was validated as "Realistic" or "Very Realistic" by 90% of Salesforce domain experts.
- Even state-of-the-art LLM agents, despite using advanced prompting and human-crafted function-calling tools, achieve low success rates (best at 64.3%) on these realistic CRM tasks, indicating a significant gap for enterprise deployment.
- Equipping stronger LLMs with human-crafted functions improved their performance, while surprisingly, weaker LLMs often performed worse with such capabilities, suggesting struggles in proper tool invocation and instruction adherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.