Automatic Workload Pinning and Regression Detection for Apache Spark

This research article discusses the challenges of managing Apache Spark version upgrades and introduces Databricks' Versionless Spark. By leveraging Spark Connect, the paper demonstrates how to decouple client applications from the Spark engine, enabling seamless upgrades and failure remediation in automated workloads. The approach minimizes disruptions while retaining programmability, offering a fully managed experience for Databricks users. The article is part of the SIGMOD-Companion 2025 conf