Reproduce “Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cen Mia Zhao, Tiantian Zhang, Hanchen Su, Yufeng Wayne Zhang, Shaowei Su, Mingzhi Xu, Yu Elaine Liu, Wei Han and 3 more
Airbnb researchers developed the Agent-in-the-Loop (AITL) framework, integrating real-time human feedback from customer support agents into a continuous learning pipeline for Large Language Models. This system drastically reduced model update cycles from months to weeks and improved retrieval precision by 14.8%, generation helpfulness by 8.4%, and agent adoption by 4.5%.
Key insights
- Live, contextual human feedback from agents significantly improves annotation quality and relevance, especially for identifying missing knowledge, compared to traditional offline methods.
- LLM-based virtual judges effectively filter low-quality annotations, maintaining data integrity and consistently improving retrieval and citation accuracy in the continuous learning pipeline.
- Retraining models using a mix of historical and newly collected annotations enhances model robustness and adaptability to both long-term patterns and recent changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.