Best Practices for Cloudera Data Engineering Deployments

This guide presents best practices for scaling Cloudera Data Engineering deployments, focusing on recommendations for managing resources effectively. It outlines the process of horizontal scaling by adding new instances or virtual clusters, while noting that vertical scaling is not supported. The document details the key components of a Cloudera Data Engineering service deployment, emphasizing the role of virtual clusters in providing isolated autoscaling compute capacity for Spark and Airflow jobs. It also discusses guidelines for scaling virtual clusters, including limits on simultaneous job submissions and recommendations for distributing job submissions over time. The guide further addresses Apache Airflow scaling and tuning considerations, including job submission rate guardrails and fine-tuning gang scheduling behavior for Spark jobs. Additionally, it provides performance tuning guidelines for Cloudera Data Engineering, such as using local SSDs and avoiding low compute profile instance types. Finally, it concludes with best practices for building Apache Spark applications, emphasizing efficient coding practices to enhance performance.