Automated Framework for Evaluating Multi-Turn Dialogues

This document is a technical report that presents an automated framework designed to evaluate the multi-turn conversational capabilities of in-car conversational assistants (ICAs). The framework addresses the challenges associated with assessing the reliability of ICAs, particularly in multi-turn interactions where users may refine their requests incrementally. It treats the ICA as a black box and utilizes closed-loop simulation with a strategy-guided user simulator, an adversarial strategy manager, and a two-tier LLM judge to assess turn-level failures and conversation-level quality. The report details the components of the framework, including the user simulator, system under test, strategy manager, and the evaluation process. The framework was evaluated on an industrial ICA, demonstrating substantial agreement with human annotators and revealing a significant increase in the detection of unique failure types. The findings underscore the necessity for systematic multi-turn testing to ensure the reliability of conversational systems in safety-critical applications.