Impact of Ungrounded LLMs on Content Discovery

This technical report examines the limitations of ungrounded large language models (LLMs) in the context of content discovery in the entertainment industry. It highlights the challenges posed by the lack of access to real-time data, which results in hallucinations and inaccuracies in the information provided by these models. The report presents findings from a study conducted by Gracenote, which analyzed the performance of ungrounded LLMs against grounded data for 2,600 titles across 13 countries. The study revealed that ungrounded LLMs achieved only moderate accuracy in matching basic content attributes, with significant variations across different regions. Notably, the report emphasizes the importance of grounding LLMs with credible external data sources to enhance their reliability. The findings indicate that a substantial percentage of responses from ungrounded LLMs were of low or zero quality, underscoring the need for improved data integration in AI-driven content discovery systems. Overall, the report outlines the critical role of accurate metadata in delivering effective user experiences in streaming services.