Managing Model Risks of Generative AI Tools in Banking

This technical report discusses the management of model risks associated with Generative AI (Gen AI) productivity tools in the banking sector. It outlines a novel validation approach tailored for these tools, which are increasingly used for tasks such as content drafting and information processing. The report emphasizes the necessity of aligning with regulatory obligations, specifically referencing SS1/23 for UK financial institutions. A structured testing framework is introduced, combining quantitative metrics and qualitative assessments to evaluate the performance of Gen AI tools across various dimensions, including accuracy and bias. The methodology involves bespoke testing using domain-specific data and automated evaluation processes. The findings advocate for a dual approach that integrates automated testing with expert reviews, enhancing the reliability of the validation process. The report concludes by highlighting the importance of continuous validation to support robust Model Risk Management practices, ensuring that Gen AI tools effectively enhance operational efficiency while mitigating risks.