Efficacy of AI RAG Tools for Information Extraction

This technical report investigates the effectiveness of AI retrieval augmented generation (RAG) tools in assisting analysts with information extraction and data annotation tasks, specifically using public disclosures from global systemically important banks (GSIBs). The study employs a within-subjects design with randomized task assignments to compare two conditions: a naive AI use condition and an interactive AI treatment condition. The findings indicate that the use of the AI tool can accelerate task execution by up to a factor of 10 and improve retrieval accuracy, particularly in the interactive condition. The report also highlights that annotator skill, both in subject matter and familiarity with AI tools, significantly influences the accuracy and speed of task performance. By replicating a real-world annotation task, the study aims to provide insights into how AI tools can enhance productivity in financial analysis, particularly in scenarios requiring quick access to unstructured data.