Comparison of Large Language Model Performance in Insurance

This technical report evaluates the performance of 19 different Large Language Models (LLMs) in the context of insurance-specific use cases. It outlines the methodology used for testing, which includes four scenarios focused on information extraction and document classification from various insurance documents. The report presents performance metrics, specifically the F1 score, which aggregates coverage and accuracy for each model across the defined tasks. It discusses the implications of model performance in relation to cost, emphasizing the importance of price/performance ratios for selecting appropriate LLMs for specific applications. The findings indicate that while newer models like Deepseek R1 and OpenAI's latest offerings show promising results, they also come with varying costs and performance characteristics. The report concludes that the distinction between high-cost and low-cost models is becoming less clear, with some lower-cost models demonstrating competitive performance. Additionally, it highlights the challenges of latency and output quality that may affect the practical deployment of these models in production environments.