Benchmark Study on Knowledge Graphs and LLM Accuracy

This technical report evaluates the accuracy of Large Language Models (LLMs) in answering enterprise questions related to SQL databases, particularly focusing on the role of Knowledge Graphs (KGs) in enhancing this accuracy. The study introduces a benchmark that includes an enterprise SQL schema specific to the insurance domain and a set of 43 natural language questions varying in complexity. The findings indicate that LLMs, when prompted with zero-shot queries directly on SQL databases, achieve an accuracy of 16%. However, when these questions are posed over a Knowledge Graph representation, the accuracy improves significantly to 54%. The report outlines the framework for the benchmark, which includes a contextual layer with an ontology that maps business concepts and relationships. This benchmark serves to facilitate the reproduction of results in various enterprise settings, thereby addressing the gaps in existing Text-to-SQL benchmarks that often overlook the complexities of enterprise data. The report concludes that investing in Knowledge Graphs can substantially improve the performance of LLM-powered question answering systems.