Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support

Abstraction: Live human-feedback flywheel continuously improving LLM customer support system

Key points:

• AITL framework integrates four annotation types directly into live customer operations: (1) pairwise response preferences, (2) agent adoption + rationales, (3) knowledge relevance checks, (4) missing knowledge identification
• Unlike batch-annotation approaches, feedback flows back in near-real-time, shrinking retraining cycles from months to weeks
• Production pilot (US-based customer support): +11.7% recall@75, +14.8% precision@8 in retrieval accuracy
• Generation quality improved +8.4% helpfulness; agent adoption rate increased +4.5%
• Demonstrates value of embedding human-in-the-loop feedback directly into operational workflows
• arXiv:2510.06674

Connections: Arxiv · AI Agents · Large Language Models · Human Feedback · Retrieval Augmented Generation

Source: https://arxiv.org/abs/2510.06674