Distributed Training on Google Cloud: Data and Model Parallelism

GCP Study Hub
September 9, 2026

Scaling model training across multiple accelerators comes down to a clear architectural choice between data parallelism and model parallelism. Data parallelism splits the training dataset across multiple devices that each hold a full copy of the model, while model parallelism divides the model itself across devices when it is too large for a single accelerator. On the Professional Machine Learning Engineer exam, selecting between these approaches depends on whether the limiting factor is how fast data can move through a model that already fits on one GPU, or whether the model exceeds single-device memory capacity.

Data parallelism for models that fit on one accelerator

Data parallelism divides the training dataset across multiple accelerators, and each accelerator holds an identical, complete copy of the model architecture and weights. Every device processes a different slice of the dataset while running the same model. This approach works best when the model fits comfortably in memory on a single accelerator. Once a model can reside on one GPU without hitting memory limits, the bottleneck is no longer model size but how much training data you can push through it. Data parallelism addresses that throughput bottleneck directly.

In a data-parallel workflow, the full training dataset is split into shards rather than sent to a single accelerator. Each shard is assigned to a separate GPU worker. For example, GPU worker 1 receives data shard 1 while holding a full copy of the model, and GPU workers 2 and 3 do the same with their own shards. Each worker processes its slice of data independently using the same model before its results are combined with the other workers.

Handling aggregation and scaling in data parallelism

Because each worker computes results on its own shard of data, those results must be combined into a single update after each step. Managing that aggregation directly on GPU workers can tie them up in communication overhead instead of computation. Reduction Server speeds up aggregation by handling it on separate VMs tuned for high network throughput. When each GPU worker finishes processing its shard, its results flow to the Reduction Server, which combines them into one update while letting the GPU workers stay focused on computation.

Implementation choices inside the training framework also affect how well data parallelism scales across machines. When using PyTorch, Distributed Data Parallel runs a separate process per GPU and scales better across multiple nodes than the older Data Parallel approach. Data Parallel relies on a single process to manage all GPUs, which often becomes a bottleneck as more GPUs are added.

Model parallelism for models that exceed single-device memory

Model parallelism takes the opposite approach from data parallelism. Instead of duplicating the model across every device, it divides the model itself and spreads it across multiple accelerators by assigning different layers or sections of the network to different GPUs. Each accelerator holds and computes only a portion of the full model. This technique is suited for very large models, such as networks with billions of parameters, that exceed the memory capacity of a single accelerator. When a model cannot fit on one GPU, tuning the batch size will not solve the problem because the model itself must be distributed, not just the training data.

Consider a model with six layers that needs to train on hardware where a single accelerator only has enough memory to hold three layers. On one accelerator, the full model does not fit and training cannot proceed. Under model parallelism, the first three layers are placed on accelerator 1 and the remaining three layers are placed on accelerator 2. The full model now fits across the two devices, with data flowing from the first accelerator to the second as it moves through the network.

Pipeline parallelism and tensor parallelism

Model parallelism is commonly implemented in two ways, depending on whether the model is split across layers sequentially or within individual layers in parallel:

  • Pipeline parallelism organizes model layers into sequential stages across devices and feeds small batches of training data through those stages. Passing smaller batches through the pipeline reduces idle time because stages can stay active on consecutive batches rather than waiting for one batch to pass through the entire network before the next one begins.
  • Tensor parallelism splits the operations of a single layer across multiple devices so they compute in parallel instead of in sequence. Rather than waiting for one device to finish an entire layer's computation, multiple accelerators share the workload of that same layer at the same time.

Matching the parallelism strategy to the bottleneck

When we evaluate distributed training scenarios on the Professional Machine Learning Engineer exam, the first check is whether the model fits in memory on a single accelerator. If it fits on one GPU and the goal is to process a large training dataset faster, data parallelism is the right match, supported by Reduction Server for aggregation and PyTorch Distributed Data Parallel for multi-node scaling. If the model is too large to fit on a single accelerator even after batch size tuning, model parallelism is required, either through pipeline parallelism across sequential layer stages or through tensor parallelism within individual layers.

Our Professional Machine Learning Engineer course covers distributed training and parallelism techniques alongside training memory errors and infrastructure optimization, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow