SWE-Lancer Benchmark for Freelance Software Engineering Tasks

This document is a technical report introducing SWE-Lancer, a benchmark comprising 1,488 freelance software engineering tasks sourced from Upwork, collectively valued at $1 million USD. The benchmark evaluates the performance of frontier language models on both Individual Contributor (IC) Software Engineering tasks and SWE Management tasks. IC tasks involve generating code patches for real-world issues, while management tasks require models to select the best implementation proposals from freelancers. The evaluation methodology includes end-to-end tests that have been triple-verified by experienced engineers, ensuring a rigorous assessment of model capabilities. The report discusses the advantages of SWE-Lancer over existing benchmarks, including its focus on real-world payouts, comprehensive grading through end-to-end tests, and the diversity of tasks that span various software engineering domains. Additionally, it highlights the economic implications of AI model development by mapping model performance to monetary values, thereby facilitating future research in automated software engineering and agentic safety.