Comparison of Large Language Models in Insurance

This technical report presents a comparative analysis of various Large Language Models (LLMs) used in the insurance sector, focusing on their performance in specific use cases. The report builds on findings from a previous study and introduces eight new LLMs while retiring two older models. It details the methodology employed by data scientists to evaluate the models across four test scenarios, including information extraction from different language invoices and document classification. The report emphasizes the importance of context size and prompt engineering in determining model performance. It includes a new table that highlights the F1 score for each model, aggregating coverage and accuracy metrics. The analysis reveals that while some models excel in certain tasks, performance can vary significantly based on the specific data fields being analyzed. The findings aim to assist stakeholders in making informed decisions regarding the selection of LLMs for various insurance applications.