Benchmark for Knowledge Graphs and LLM Accuracy

This technical report evaluates the accuracy of Large Language Models (LLMs) in answering enterprise questions related to SQL databases, particularly in the insurance domain. It introduces a benchmark that includes an enterprise SQL schema, a variety of enterprise queries, and a contextual layer that utilizes a knowledge graph. The study finds that LLMs, when prompted with zero-shot questions directly on SQL databases, achieve an accuracy of 16%. However, this accuracy improves significantly to 54% when the questions are posed over a knowledge graph representation of the SQL database. The report outlines the importance of integrating knowledge graphs to enhance the accuracy of LLM-powered question answering systems, thereby reducing the risk of inaccuracies and improving trustworthiness in enterprise applications. The findings suggest that investing in knowledge graphs can lead to better outcomes in LLM-based question answering systems.