Search-based Testing Approach for In-Car Scene Understanding

This technical report presents ISU-Test, an automated testing approach designed for evaluating in-car scene understanding (ISU) systems that utilize vision-language models (VLMs). The report outlines the challenges associated with traditional testing methods, particularly the difficulties in collecting real-world in-vehicle data during early design stages. ISU-Test combines rendering-based scene generation with search-based optimization to systematically identify failures in VLM outputs. The methodology includes defining scene features, parametrizing scenes, and employing search-based optimization to generate diverse scenarios. The report evaluates ISU-Test against both an industrial prototype and open-source VLMs through two case studies: visual question answering and captioning. Results indicate that ISU-Test significantly outperforms random scenario generation, achieving higher failure rates and improved coverage. The findings emphasize the importance of rigorous testing for ensuring the reliability of ISU systems, which are increasingly mandated by regulatory frameworks for driver monitoring functions.