Reproduce “Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aman Gupta, Kevin Rossell, Edesio Alcobaça, Jose Chrystian Lima Pacheco, Carolina Baptista de Lima, Shao Tang, Luiz Paulo Rabachini, Luis Moneda and 3 more
The system helps support agents handle card-delivery cases by modularizing instructions, routines, tools, and memory. It tests candidate changes against expert-labeled failures, then compares offline evaluator scores with deployed customer metrics. In card delivery, AI tNPS rose 37 percentage points and SSR 29 percentage points; these version-level gains accompanied lower evaluator failure rates but do not establish causation.
Key insights
- The quality and reliability of the evaluation pipeline directly determine the speed and confidence of iteration in AI agent development.
- Offline evaluation metrics, when rigorously designed and calibrated, can reliably predict positive impacts on online business metrics for large-scale customer support agents.
- A modular prompt architecture combined with robust tool design is essential for building compliant, adaptable, and high-quality AI agents in high-stakes operational environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.