
Managed Service for Apache Spark (formerly Dataproc) is Google Cloud's on-demand service for Apache Hadoop and Apache Spark. It is built for processing large amounts of data with these familiar tools, without the complexity of managing the underlying infrastructure. The name is commonly shortened to Managed Spark, so it is worth recognizing both forms. On the Professional Cloud Architect exam, it is the go-to service for on-premises Apache Spark or Hadoop jobs that are moving to Google Cloud.
With Managed Spark, you can quickly set up a Hadoop cluster to handle a range of data processing tasks, from batch processing to real-time analytics. The service integrates with other Google Cloud services, which makes it straightforward to build scalable and cost-effective data pipelines. Because the cluster infrastructure is managed for you, the focus shifts from operating the cluster to getting insights from the data.
A Managed Spark cluster follows the distributed computing model, with a master node coordinating a set of worker nodes.
The master node acts as the central coordinator and runs two components:
Each worker node also runs two components:
The name node and the data nodes continuously exchange metadata, so the name node knows where each block of data is located and the data nodes can report back on their status. In the same way, the resource manager delegates tasks to the node managers, which carry them out on their own workers.
A cluster is not limited to a handful of workers. The number of worker nodes can scale significantly based on the size of the data and the complexity of the workload, and it can scale up or down with demand, which is part of the elastic nature of cloud-based clusters like Managed Spark.
This design brings two benefits. Splitting the workload across many nodes, instead of relying on one central processor, allows parallel processing, so large jobs complete faster. And because both data and tasks are distributed, with data replicated across nodes, the cluster avoids bottlenecks and single points of failure. If one node fails, the data remains available on the others.
MapReduce is one of the processing frameworks that Managed Spark supports, and it is the foundation of Hadoop workloads. It uses the cluster architecture directly, with the master node coordinating tasks and the worker nodes executing them in parallel. A MapReduce job runs in two phases:
HDFS, the Hadoop Distributed File System, is the storage side of the same ecosystem. It stores large volumes of data by distributing it across multiple nodes. Because HDFS uses block-based storage, large files are split into smaller blocks that are spread across the data nodes. That lets processing tasks run in parallel close to where the data is stored, which reduces data transfer between nodes and lowers latency. Data nodes also replicate blocks to other data nodes, so the system can keep functioning when a node goes down. In this model, HDFS provides the storage, while MapReduce and Spark provide the computation.
Apache Hive is an Apache data system that was originally designed for running SQL-like queries on large datasets stored in Hadoop, in the manner of a data warehouse. Hive is commonly present in workloads that are being migrated to the cloud, which means the data associated with on-premises Hive systems has to be migrated as well. The most common formats for Hive data are Parquet and ORC (Optimized Row Columnar).
Managed Spark is built to run Hive queries efficiently on managed Hadoop clusters, handling the cluster infrastructure while the queries run. Hive data is usually stored in Cloud Storage, where Managed Spark can access it. From Cloud Storage, the data can be pulled into Managed Spark using the Cloud Storage connector, or it can be queried directly in BigQuery through external tables. That gives an architect a choice of tool for the same data: Hadoop-style processing with Managed Spark, or analytics through BigQuery.
The rule of thumb centers on existing Hadoop and Spark jobs. If an organization has on-premises Apache Spark or Hadoop jobs and is moving them to Google Cloud, those jobs should be migrated to Managed Spark, which is the go-to cloud-based service for these workloads. The jobs keep running on the same open source frameworks, while Google Cloud takes on the work of managing the cluster infrastructure. That rule is worth having firmly in place for the Professional Cloud Architect exam.
Our Professional Cloud Architect course covers Managed Service for Apache Spark alongside Dataflow and BigQuery, with practice questions that drill these distinctions.