Fine-Tuning LLM Models for Information Extraction in Insurance

This technical report presents the findings of the Shift Technology data science and research teams on the performance of fine-tuned large language models (LLMs) for information extraction tasks in the insurance sector. The report outlines the methodology used to evaluate three models: GPT4o, GPT4o-Mini, and a fine-tuned version of GPT4o-Mini across various test scenarios, including English-language airline invoices, Japanese-language property repair quotes, and French-language dental invoices. The evaluation metrics focused on coverage and accuracy, with performance reported as an F1 score. The results indicate that fine-tuning can enhance LLM performance, particularly when applied to complex datasets. However, the report also notes that fine-tuning is not universally beneficial and may lead to performance degradation in different use cases. The cost implications of fine-tuning are discussed, highlighting its potential as a cost-effective strategy for improving LLM performance in insurance applications.