Apache Spark Deployment & Administration

Apache Spark is an open-source, distributed data processing engine designed to enable the rapid processing of large datasets across clusters of machines. It supports batch processing, SQL analytics, streaming, machine learning, and graph processing.

Apache Spark supports many cluster managers, such as Spark Standalone, Apache Hadoop YARN, and Kubernetes. YARN offers resource management and scheduling for Spark applications within Hadoop environments.

Let’s learn Apache Spark configuration, deployment, and administration in detail in this blog.

Table of Contents

• Apache Spark Overview
• Spark on Kubernetes – A Close Look
• Spark Deployment Modes Comparison
• Spark Connect Architecture
• Spark Security Configuration
• Spark Resource Administration
• Why run Spark on YARN?
• Spark Performance Concepts
• FAQs
• Conclusion

Apache Spark - An Overview

Spark enables you to perform computations across multiple machines. It enables organizations to handle datasets that are too large or computationally intensive for a single system.

As of August 2026, Apache Spark 4.2.0 is the latest stable release. Let’s explore Apache Spark in more detail.

Key Highlights

• Kubernetes has become the primary way to deploy Spark in cloud-native environments.
• Kubernetes is a first-class deployment option alongside YARN.
• Spark 4.0 and later require Java 17 or later.

Key Features

Let’s explore the key features of Apache Spark below.

1. Spark Connect

The client-server architecture separates the Spark client from the cluster. Instead of using the full PySpark, which is 355 MB, a lightweight Python client, pyspark-connect, is used to connect to a remote Spark cluster through the DataFrame API.

It enables IDE-native development, such as in PyCharm or Jupyter, when working with production clusters. You can use the spark.api.mode configuration option to enable or disable Spark Connect functionality.

2. Structured Logging

This feature causes log output to be in JSON format by default, with the setting being spark.log.structuredLogging.enabled=true. JSON logs can be directly integrated with log aggregation systems such as ELK, Grafana Loki, and Splunk.

3. ANSI Mode Enablement

In Apache Spark 4.2.0, the ANSI SQL mode is enabled by default using the setting spark.sql.ansi.enabled=true.

When ANSI mode is enabled, Spark adheres to more rigorous SQL semantics. It causes Spark to report errors for operations that would otherwise produce unexpected or invalid results.

4. Adaptive Query Execution (AQE)

In Apache Spark 4.2.0, Adaptive Query Execution (AQE) dynamically optimizes queries during execution. Rather than relying entirely on estimates generated during query planning, AQE uses runtime statistics to modify the physical execution plan.

Key refinements in AQE include:

• AQE can combine small shuffle partitions to reduce task overhead.
• AQE identifies skewed, shuffled partitions and adjusts execution to minimize the effects of uneven data distribution.
• It can adjust some join strategies during execution if the actual statistics differ from the original estimates.
• Runtime Information helps Spark make better optimization decisions.
• Partition management has been improved, as AQE can reduce unnecessary shuffle partitions, thereby improving resource utilization.

5. Large Workload Stability

Apache Spark ensures stability for large workloads with plans for zstd-compressed protobufs, chunked Arrow result streaming, and a RocksDB state store provider featuring improved lock management. Improvements help Spark handle large Structured Streaming workloads more reliably.

6. GCE/Cloud Deployment

Google Cloud Dataproc is a managed service that lets you run Apache Spark and other open-source data-processing frameworks on Google Cloud. It helps you set up, configure, scale, monitor, and integrate clusters with Google Cloud services.

Google Cloud Dataproc is a service for Spark and Hadoop that manages cluster provisioning, configuration, and patching. Dataproc automatically supports every new Spark release. Google Cloud Dataproc offers specific image versions that include certain Spark versions.

Spark on Kubernetes – A Close Look

With Apache Spark 4.2, Spark can run directly on Kubernetes, with driver and executor pods being scheduled by Kubernetes. The current Spark 4.2 documentation states that Kubernetes 1.34 or later is the prerequisite.

• Deployment Mode – Spark Submit offers both cluster and client modes when used with Kubernetes; cluster mode is usually simpler because the driver program runs within the cluster.
• Driver and Executors – The driver runs within a Kubernetes pod and creates executor pods. Kubernetes then schedules both pods based on available resources and scheduling constraints.
• Container Images – Spark applications need container images; you can use the official “apache/spark:<version>” image or build your own image, which includes the application dependencies.
• Kubernetes API/RBAC – The driver service account should have the necessary permissions to create and monitor executor pods, services, and ConfigMaps. You can use namespace-scoped RBAC whenever possible.

Apache Spark Deployment Modes Comparison

You can deploy Apache Spark using standalone, Kubernetes, and YARN deployment methods. The table below compares them based on multiple factors:

Spark Deployment on Kubernetes

You need to use the k8s:// master URL when running spark-submit for Spark applications on Kubernetes. The following bash command demonstrates this:

The Spark driver runs in a Kubernetes pod and uses the Kubernetes API to create executor pods. When running in cluster mode, the application code and its dependencies must be accessible to the driver through a container image or a remote dependency location.

Creating Docker Images

Spark provides the docker-image-tool.sh script to build Spark Docker images. The following example demonstrates how to build the image:

Next, you need to push the image to a container registry accessible by Kubernetes.

When deploying Spark in a production environment, you need to customize the image to include the application's dependencies, required Python or Java libraries, certificates, and other runtime requirements.

Spark Connect Architecture

Spark Connect is a client-server architecture that separates the Spark client from the Spark driver. The client transmits logical plans to a Spark Connect server via gRPC. The server executes the plans using Spark's driver and cluster resources.

The architecture consists of multiple components, including a Spark Connect client, a gRPC connection, a Spark Connect server, and a Spark cluster.

• A Spark Connect client – It runs within a user's application, notebook, IDE, or development environment.
• The gRPC connection – It passes requests and Spark protocol messages between the client and the server.
• The Spark Connect server – It runs alongside the Spark driver and accepts logical plans from clients.
• The Spark cluster – It runs the application using the configured executors and cluster manager.

Starting a Spark Connect Server

First, start the Spark Connect server using the standard Spark installation.

./sbin/start-connect-server.sh

You can specify Spark configuration properties when starting the server. The default Spark Connect port is 15002.

Spark Security Configuration

You can configure Apache Spark security according to the deployment environment. A production setup typically combines TLS, authentication, authorization, encryption, secret management, and network controls to ensure a secure configuration.

SSL/TLS Configuration

Spark supports SSL/TLS for its network endpoints, including the Spark UI and internal Spark communication. You can configure the appropriate SSL and keystore/truststore settings for the services you need to protect.

Below is an example:

If you use the Spark Web UI, you can configure the UI-specific SSL settings using the script below.

spark.ssl.ui.enabled=true

Best Practices in Configuration Security

Below are some best practices you must follow to ensure a secure configuration.

• Use TLS for externally accessible Spark services and for sensitive network paths
• Use Kerberos when Spark operates on a secure YARN/HDFS infrastructure
• Authorization for Spark-on-Kubernetes should use Kubernetes authentication and RBAC
• Use an identity infrastructure based on OIDC or OAuth for user-facing gateways or for authenticating with Kubernetes.
• Use Access Control Lists (ACLs) in Spark to limit access to the application or user interface.
• Use ServiceAccounts with the minimum required privileges for Kubernetes workloads.

Apache Spark Resource Administration

Apache Spark can run on Apache Hadoop YARN, which manages and schedules cluster resources.

1. Applications

A Spark application includes a driver program and executor processes; the driver coordinates the application, and the executors perform tasks and store data. Spark jobs are broken down into stages, which include tasks.

2. Structural Planning of a Spark Application

A Spark application includes a driver program that creates and manages a Spark session or Spark context; the driver coordinates the executors and schedules tasks across the cluster.

An application can be used for a single group of work, an interactive session with different tasks distributed across the cluster, or a persistent server continuously fulfilling requirements.

Unlike MapReduce, a process will have procedures, called Executors, running on the batch for its sake when it’s not running any tasks. This methodology enables in-memory data caching for fast access and extremely fast task startup times.

3. Executors

MapReduce runs every job in its own process. Spark executors are processes launched on worker nodes by the cluster manager. They perform tasks, cache or persist data, and communicate with the driver. 

Generally, executors remain available for the entire duration of an application, provided that the executor lifecycle and dynamic allocation settings allow it.

This process remains active for the lifetime of the Spark application, even when no jobs are running. The advantage of this model is the speed at which it completes the process. Jobs can start up rapidly and process in-memory data. The disadvantage of this model is coarse-grained resource management.

When dynamic allocation is not in use, the number of executors assigned to an application usually stays the same after they have been allocated. However, when dynamic allocation is enabled, Spark can add or remove executors in response to workload demand.

4. Spark Driver

To manage the task stream and schedule tasks, Spark relies on a dynamic driver process.

The Spark driver is responsible for coordinating application execution and scheduling tasks in client-deploy mode. Conversely, in MapReduce, the client process can exit while the task continues running.

In Hadoop 1.x, the JobTracker was responsible for job scheduling, and in Hadoop 2.x, the MapReduce client process is responsible for the same.

5. Pluggable Resource Management

Spark supports pluggable cluster managers. The cluster manager is responsible for launching executors. Spark application developers do not need to manage cluster-manager-specific details.

Spark is compatible with various cluster managers, such as Spark Standalone, Apache Hadoop YARN, and Kubernetes. Each of these systems has two components. A cluster manager (the YARN Resource Manager or a Spark Standalone client) chooses which applications run, where, and when.

Why Run Spark on YARN?

Using YARN as Spark’s cluster manager provides several advantages over Spark Standalone:

• YARN allows you to actively share and manage the same pool of cluster resources across workloads.
• YARN enables various workloads, such as Spark and MapReduce applications, to use cluster resources under a central resource management system.
• Spark Standalone mode requires every application to run an executor on every node in the group, while with YARN, you pick the number of executors to utilize.
• YARN is the main cluster manager for Spark that provides strong security capabilities. With YARN, Spark can keep running against Kerberized Hadoop clusters.

Executing on YARN

While MapReduce launches a container and starts a JVM for each task, Spark runs tasks within the same container. Spark applications can run on YARN in two deployment modes: In client mode, the driver operates within the client process that submits the application. In cluster mode, the driver operates within the YARN cluster.

Cluster mode is commonly used for production applications because the driver is managed within the cluster rather than relying on the submitting client to remain available. Understanding this distinction requires familiarity with YARN’s application client concept.

In YARN, every application has an Application Client process, which is the first process started for that application. The application is responsible for requesting resources from the Resource Manager and, when allocated, instructing NodeManagers to start containers.

In cluster mode, the driver runs in the ApplicationMaster. This implies that the same procedure both drives the application and requests resources from YARN, and that it runs inside a YARN container. The client that begins the application doesn’t have to remain connected for its whole lifetime.

In application mode, however, it is not appropriate for interactive Spark use. Spark applications that require client information, such as Spark Shell and PySpark, need the Spark driver to remain running within the client process that starts the application.

In yarn-client mode, the Application Master is present to request cluster manager containers from YARN.

Key Spark Performance and Execution Concepts

We’ll next walk through the key Spark performance and execution concepts in the following.

Key Components

We’ll first review the major components of Spark's architecture, including caching, shuffling, serialization, and partitioning.

1. Caching

Spark provides a feature that enables frequently reused datasets to be cached or persisted, so that subsequent operations can avoid recomputing them. The different persistence levels specify whether the data is stored in memory, on disk, or using both memory and disk.

However, it will be challenging if you have multiple cached datasets, as the amount of resources depends on both the chosen storage level and the dataset size.

2. Shuffle

Some operations require data movement, although shuffling can be expensive. For instance, by shuffling data, it is redistributed among the partitions, usually among the executor nodes, so that the records required by subsequent operations are colocated.

3. Serialization

The easiest way to achieve serialization is to use different code on the same data. It has led to the use of the JVM across many frameworks.

Serialization involves converting objects or data structures into a byte-oriented format. It allows them to be stored or transmitted efficiently. Spark uses serializers such as Java serialization and Kryo.

4. Partitions

Spark includes many programming abstractions such as RDDs and the more advanced DataFrame and Dataset APIs. When it comes to processing structured data, DataFrames and Spark SQL are usually preferred to the direct use of RDDs.

Complex operations such as groupBy or join can be performed using Spark’s DataFrame and Dataset APIs.

Spark Performance Optimization

We will now discuss how these components affect Spark application performance.

1. Memory Management and Caching

Spark supports in-memory caching. A disadvantage of memory caching is that it consumes significant memory. If you use YARN and the JVM, they consume a significant amount of memory. This reduces the amount of memory available for other operations, such as caching and data movement.

Therefore, it is important to use appropriate partitioning strategies. It will help reduce memory management and improve resource utilization. It helps avoid data skew.

Moreover, you can use batches to divide long-running jobs. It allows you to run each batch of the job in a new environment with no accumulated metadata.

2. Minimize Unnecessary Data Movement

Minimizing data transfer and avoiding shuffles would lead to faster and more reliable programs. Repartition() should be used when it is necessary to redistribute the data across the partitions, and coalesce() should be used when you want to reduce the number of partitions without needing a full shuffle.

It is advisable to avoid unnecessary repartitioning, as shuffles can increase both network and disk I/O. You should also avoid join operations like join and cogroup, as well as ByKey operations such as reduceByKey and groupByKey. Two mechanisms provided by Spark are Broadcast variables and Accumulators.

3. Speed

Caching should be employed only when a dataset is used across multiple stages, and recomputation would be expensive. Instead of using data twice, cache it in memory.

Broadcast variables should be used for small, read-only datasets that are reused across different tasks and would otherwise demand costly data shuffles. When broadcasting large values, it is necessary to monitor executor memory usage.

Finally, you should run Spark programs in parallel. Processing one key at a time could lead to poor resource utilization. Using this method would also deny you the advantages of Spark's built-in parallelism.

Frequently Asked Questions

1. What are Spark deployment modes?

Client mode and cluster mode are the two primary deployment modes of Apache Spark.

2. Is Mesos supported in Spark 4?

No. Apache Mesos is not supported in Apache Spark 4.x.

3. What is Spark Connect?

Spark Connect is an Apache Spark client–server architecture that separates the application client from the Spark driver. It was introduced in Spark 3.4 and is still supported in Spark 4.x.

4. What Java version does Spark 4 require?

Apache Spark 4.x requires at least Java 17 and officially supports Java 17 and 21 as of Spark 4.2.0.

5. YARN vs Kubernetes for Spark — which should I use?

Both YARN and Kubernetes are supported as cluster managers for Spark 4.2. The choice largely depends on your existing platform and operational requirements.

6. What changed in Spark 4.0 logging?

The major logging enhancement in Spark 4.0 is optional structured logging. This new version includes support for JSON Template Layout when emitting logs, making them simpler to parse, search, correlate, and feed into modern observability platforms.

Conclusion

This tutorial has provided you with an in-depth understanding of Apache Spark deployments, configurations, and Apache Spark Connect architecture. You have gained an understanding of Spark security concepts, running Spark on YARN, and much more.

If you would like to learn more about Apache Spark and its core concepts, you are not alone. You can step into the MindMajix Apache Spark Course to further advance your career.