Microsoft Copilot for Security Randomized Controlled Trial

This whitepaper presents the findings from randomized controlled trials (RCTs) conducted to evaluate the efficiency gains from using Microsoft Copilot for Security within a Microsoft Defender XDR environment. The trials involved two groups of subjects: one with access to Copilot and another without. The tasks performed included Incident Summarization, Script Analysis, Incident Report, and Guided Response. Results indicated that security professionals using Copilot achieved higher accuracy and quality in their outputs, with significant improvements noted in various tasks. For instance, Copilot users were found to be 7% more accurate overall and completed tasks 22% faster. Additionally, the paper details the performance of security novices, who also showed marked improvements in accuracy when using Copilot. The findings suggest that Copilot enhances productivity and quality of work across different user skill levels, highlighting its potential impact on security operations.