Dataflow RunInference API: In-Pipeline ML Inference on Google Cloud

GCP Study Hub
September 24, 2026

The Dataflow RunInference API is an Apache Beam transform that runs machine learning predictions directly inside a Dataflow pipeline. Rather than sending each incoming element over the network to a separate prediction service, the model executes in the same pipeline step that is already processing your data. Removing that network round trip keeps prediction latency very low. For the Professional Machine Learning Engineer exam, understanding how RunInference handles execution, batching, model updates, and hardware acceleration helps separate in-pipeline inference from architectures that rely on external endpoints.

Low-latency execution and dynamic batching

Because the RunInference transform runs inside the Dataflow pipeline step itself, elements do not need to leave the pipeline for a remote prediction call. Eliminating that network round trip is what keeps prediction latency low.

To improve throughput, RunInference also dynamically batches incoming inputs. Instead of invoking the model once for each individual element, Apache Beam groups incoming elements together and passes them to the model as a batch. Beam tunes the batch size automatically based on the load the pipeline is seeing. Larger batches make more efficient use of model compute, especially when inference runs on accelerators.

Automatic model refreshes with no downtime

When models are retrained on a schedule, the RunInference transform automatically refreshes the model so the pipeline always uses the latest deployed version. You publish a new model artifact, and the running pipeline picks it up and swaps to the updated version in place.

Because the refresh happens without stopping the Dataflow job or redeploying the pipeline, workloads that rely on scheduled retraining can run daily model updates with no downtime.

GPU acceleration on Dataflow worker VMs

RunInference can use GPUs on supported runners to increase inference speed. Dataflow lets you attach GPUs to worker VMs so the prediction step runs on accelerated hardware. This setup pays off for deep learning models where the computation is heavy enough to justify the accelerator. Enabling GPU acceleration on the workers speeds up inference without requiring changes to how the pipeline itself is written.

How RunInference works in a streaming pipeline

To see how these pieces fit together, consider a streaming fraud detection job running on Dataflow:

  • Input data arrives from a streaming source, such as a stream of transaction events read from Pub/Sub.
  • Inside the pipeline, the RunInference transform owns the ML model and handles predictions for every element flowing through.
  • The model is loaded into memory once per worker process rather than once per element, and it refreshes automatically whenever a new version becomes available.
  • Each transaction event passes through the model and comes out as a prediction, with optional GPU acceleration on the worker VMs speeding up the entire path without changing the pipeline code.

On the Professional Machine Learning Engineer exam, scenarios that combine streaming ingestion from Pub/Sub, low-latency predictions without external network round trips, scheduled model updates without stopping the job, and optional GPU acceleration for deep learning models point directly to Dataflow with the RunInference API.

Our Professional Machine Learning Engineer course covers the Dataflow RunInference API alongside Dataflow streaming pipelines and Pub/Sub ingestion, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow