Challenges in Using Large Language Models for Code Repair

This technical report discusses the challenges associated with using large language models (LLMs) in automated code generation and program repair. It outlines the increasing integration of AI-based coding assistants in software development, noting that a significant percentage of developers utilize these tools. The report highlights the advancements in automated program repair (APR) facilitated by generative AI models, which have shown promise in fixing software defects. However, it emphasizes the limitations of LLMs, particularly in addressing complex security-related issues and generating code that meets various functional and non-functional requirements. The report also addresses concerns regarding the quality and security of AI-generated code, stressing the need for scrutiny and validation. Furthermore, it discusses the insufficiency of training data for LLMs to effectively distinguish between vulnerable and secure code, indicating that supervised approaches may be necessary for successful APR. The report concludes by suggesting that human oversight is essential in the repair process, especially for security misconfigurations.