Managed Service for Apache Spark (formerly Dataproc): An Overview for Architects

GCP Study Hub
November 5, 2026

Managed Service for Apache Spark (formerly Dataproc) is Google Cloud's on-demand service for Apache Hadoop and Apache Spark. It is built for processing large amounts of data with these familiar tools, without the complexity of managing the underlying infrastructure. The name is commonly shortened to Managed Spark, so it is worth recognizing both forms. On the Professional Cloud Architect exam, it is the go-to service for on-premises Apache Spark or Hadoop jobs that are moving to Google Cloud.

What the service does

With Managed Spark, you can quickly set up a Hadoop cluster to handle a range of data processing tasks, from batch processing to real-time analytics. The service integrates with other Google Cloud services, which makes it straightforward to build scalable and cost-effective data pipelines. Because the cluster infrastructure is managed for you, the focus shifts from operating the cluster to getting insights from the data.

How a Managed Spark cluster is organized

A Managed Spark cluster follows the distributed computing model, with a master node coordinating a set of worker nodes.

The master node acts as the central coordinator and runs two components:

  • The HDFS name node, which manages the metadata that describes how and where data is distributed across the cluster, essentially a map of where each file is stored.
  • The YARN resource manager, which orchestrates computational resources and allocates each task to the available worker nodes, so work runs in parallel and resources are well used.

Each worker node also runs two components:

  • The HDFS data node, which handles the actual storage of data and replicates it across other nodes.
  • The YARN node manager, which executes the tasks assigned by the resource manager locally, so computation happens as close to the data as possible and less data has to move between nodes.

The name node and the data nodes continuously exchange metadata, so the name node knows where each block of data is located and the data nodes can report back on their status. In the same way, the resource manager delegates tasks to the node managers, which carry them out on their own workers.

A cluster is not limited to a handful of workers. The number of worker nodes can scale significantly based on the size of the data and the complexity of the workload, and it can scale up or down with demand, which is part of the elastic nature of cloud-based clusters like Managed Spark.

This design brings two benefits. Splitting the workload across many nodes, instead of relying on one central processor, allows parallel processing, so large jobs complete faster. And because both data and tasks are distributed, with data replicated across nodes, the cluster avoids bottlenecks and single points of failure. If one node fails, the data remains available on the others.

MapReduce and HDFS

MapReduce is one of the processing frameworks that Managed Spark supports, and it is the foundation of Hadoop workloads. It uses the cluster architecture directly, with the master node coordinating tasks and the worker nodes executing them in parallel. A MapReduce job runs in two phases:

  • In the map phase, the input data is split into smaller chunks, distributed across nodes, and processed by mappers. Each mapper works on its piece of the dataset in isolation, performing operations such as filtering or transformation.
  • The reduce phase begins with a shuffle step, where the outputs of the mappers are reorganized and sorted to prepare them for the next step. Reducers then combine the processed data into a final aggregated result.

HDFS, the Hadoop Distributed File System, is the storage side of the same ecosystem. It stores large volumes of data by distributing it across multiple nodes. Because HDFS uses block-based storage, large files are split into smaller blocks that are spread across the data nodes. That lets processing tasks run in parallel close to where the data is stored, which reduces data transfer between nodes and lowers latency. Data nodes also replicate blocks to other data nodes, so the system can keep functioning when a node goes down. In this model, HDFS provides the storage, while MapReduce and Spark provide the computation.

Hive data, Cloud Storage, and BigQuery

Apache Hive is an Apache data system that was originally designed for running SQL-like queries on large datasets stored in Hadoop, in the manner of a data warehouse. Hive is commonly present in workloads that are being migrated to the cloud, which means the data associated with on-premises Hive systems has to be migrated as well. The most common formats for Hive data are Parquet and ORC (Optimized Row Columnar).

Managed Spark is built to run Hive queries efficiently on managed Hadoop clusters, handling the cluster infrastructure while the queries run. Hive data is usually stored in Cloud Storage, where Managed Spark can access it. From Cloud Storage, the data can be pulled into Managed Spark using the Cloud Storage connector, or it can be queried directly in BigQuery through external tables. That gives an architect a choice of tool for the same data: Hadoop-style processing with Managed Spark, or analytics through BigQuery.

When an architect chooses Managed Spark

The rule of thumb centers on existing Hadoop and Spark jobs. If an organization has on-premises Apache Spark or Hadoop jobs and is moving them to Google Cloud, those jobs should be migrated to Managed Spark, which is the go-to cloud-based service for these workloads. The jobs keep running on the same open source frameworks, while Google Cloud takes on the work of managing the cluster infrastructure. That rule is worth having firmly in place for the Professional Cloud Architect exam.

Our Professional Cloud Architect course covers Managed Service for Apache Spark alongside Dataflow and BigQuery, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow