Generative AI in Test Development and Psychometrics

This white paper discusses the potential applications of generative AI (GenAI) models in the field of psychometrics, particularly in test development and assessment processes. It outlines various activities where GenAI can be utilized, including job task analysis, competency development, item bank management, and statistical analysis. The report emphasizes the importance of understanding both the promising uses and the limitations of GenAI, as well as the ethical considerations involved in its application. It presents findings from a literature review and interviews with professionals in the field, aiming to provide practical recommendations for integrating GenAI into existing testing practices. The report also evaluates the performance of GenAI models, such as ChatGPT, in high-stakes credentialing exams, noting their ability to achieve passing scores and generate rationales aligned with ethical practices. Overall, the document serves as a resource for assessment professionals to navigate the emerging trends and technologies associated with GenAI in psychometrics.