Automatic Benchmark Generation for LLM Code Tasks

This technical report presents a methodology for generating and evaluating benchmarks for code-related tasks using large language models (LLMs). The authors introduce a concept termed 'LLM as a Judge' (LaaJ), which aims to assess the quality of artifacts produced by LLMs in various programming tasks, such as code translation and summarization. The report outlines the development of an automated benchmark generation engine that creates code samples across multiple programming languages. This benchmark serves dual purposes: it aids in the validation of LaaJ implementations and tests the effectiveness of LLM-generated solutions. The authors detail the process of using a graph representation to facilitate the generation of code artifacts and establish expectations for their usefulness. Additionally, the report discusses the challenges associated with evaluating LLM outputs and the importance of early testing in the development cycle. The findings contribute to the understanding of how to reliably assess the quality of LLM-generated code artifacts.