Study of Open Source Upgrade Recommendations in AI Development

This technical report presents a comprehensive analysis of 37,000 open source upgrade recommendations related to AI software development. The study evaluates the effectiveness of various AI models, including GPT-5 and newer frontier models, in making dependency decisions. It highlights the structural weaknesses of these models in providing reliable upgrade paths, often leading to wasted resources and unresolved vulnerabilities. The report details the findings from the analysis, noting that while newer models show improvements in reducing hallucinations, they still exhibit significant error rates in recommendations. Approximately one in three components receives a 'no-change' recommendation, which may preserve existing risks. The report emphasizes that the lack of real-time dependency intelligence and security context limits the models' ability to make safe remediation decisions. The findings underscore the importance of combining AI productivity with real-time software intelligence to achieve better software outcomes and mitigate risks associated with dependency management.