Microsoft Copilot for Security Randomized Controlled Trial

This whitepaper presents the findings from randomized controlled trials conducted to evaluate the efficiency gains from using Microsoft Copilot for Security. The trials involved external subjects performing tasks in a Microsoft Defender XDR environment, with one group having access to Copilot capabilities while the control group did not. The tasks included Incident Summarization, Script Analysis, Incident Report, and Guided Response. The study measured both speed and accuracy, revealing that security professionals using Copilot were 7% more accurate overall and completed tasks 22% faster. Additionally, Copilot users produced higher quality incident summaries, achieving a 49% higher content score compared to control users. The findings also indicated that security novices using Copilot were significantly more accurate in various tasks, with improvements of up to 43% in specific areas. The results suggest that Copilot enhances productivity and quality of work for both novice and experienced security analysts.