Comparison of Large Language Models in Insurance

This report presents a comparison of various Large Language Models (LLMs) specifically in the context of insurance applications. It outlines the methodology used by Shift Technology's data science and research teams, which involved testing 16 publicly available LLMs across four distinct scenarios. These scenarios included information extraction from airline invoices and property repair quotes, as well as document classification related to travel insurance claims. The report details the performance metrics of each LLM, including coverage and accuracy, and highlights the advancements in LLM technology since previous volumes. Key findings indicate that GPT4o achieved the highest aggregate performance score, while GPT4o-Mini and Claude3.5 Sonnet also demonstrated competitive results. The analysis emphasizes the importance of price/performance ratios in selecting LLMs for insurance use cases, suggesting that these factors may influence future model selection as performance levels converge across different models.