If you have worked with ML pipelines before, you know that writing the pipeline logic is only half the problem. The other half is figuring out where and how to run it. Vertex AI Pipelines exists to make that second half easier.
This article covers two things you should understand about Vertex AI Pipelines for the Professional Machine Learning Engineer (PMLE) exam: how pipeline DAGs work, and how Vertex AI Pipelines relates to Kubeflow and TensorFlow Extended.
ML pipelines automate the steps of an ML workflow, from data ingestion through training, evaluation, and deployment. That automation saves time and reduces the chance of errors. In practice, these workflows are not simple linear sequences. They include branching, conditional logic, and sometimes steps that invoke external dependencies outside the ML workflow itself.
Vertex AI Pipelines structures these workflows as directed acyclic graphs, or DAGs. Each step in the pipeline is a component (the Vertex AI term for a node in the graph), and the relationships between components are defined by directed edges.
Here is a concrete example. Suppose you are building a customer churn model. The pipeline might look like this:
Notice that each step is more specific than the broad categories you might see in a textbook diagram of an ML pipeline. And the conditional logic at step 7 creates a branch in the graph. In practice, a production pipeline might have many branches like this, and the conditions themselves can be more complex.
Three properties make this a DAG. It is directed, meaning the arrows go one way. It is acyclic, meaning there are no loops or circular dependencies. And it is a graph, meaning it is structured as nodes connected by edges.
A reasonable question comes up here: if cycles are not allowed, how do you retrain a model? The ML lifecycle looks cyclical. You train, deploy, monitor, and then retrain.
The answer is that retraining happens at the orchestration level, not within a single DAG run. When a model monitoring job detects data drift or performance degradation, it does not loop back within the same pipeline execution. It triggers a completely new pipeline run.
Using the churn model example, the pipeline might run successfully on Monday. That is DAG run 1. On Friday, monitoring detects that customer behavior patterns have shifted. This triggers DAG run 2 with fresh data. It is an entirely separate execution that starts from data ingestion again, using the same pipeline definition but with different parameters like new date ranges, updated data sources, or incremented version numbers.
Vertex AI handles this through scheduled runs, event-driven triggers, or Pub/Sub messages when new data arrives. The conceptual ML lifecycle appears cyclical, but each individual pipeline execution remains acyclic. That property is what allows Vertex AI to optimize execution order and ensure pipelines complete successfully.
To understand what Vertex AI Pipelines actually does, it helps to understand what came before it.
Before Vertex AI, ML pipelines were built and run using Kubeflow Pipelines. The workflow had two layers. First, you defined your pipeline in Python using the Kubeflow Pipelines SDK. This is where you specified each step (preprocessing, training, evaluation) and how those steps connected together. Second, the Kubeflow Pipelines orchestration layer compiled that definition and executed the steps in the right order using containerized workloads.
The catch was that Kubeflow ran on Kubernetes. That is where the name comes from. Running it meant standing up and maintaining Kubernetes clusters, managing networking and compute resources, and making sure everything scaled and stayed active. It was a big improvement at the time, but it required significant Kubernetes knowledge just to run ML workflows. That created a barrier for many teams.
TFX is a library of standardized components for building production ML workflows, particularly for TensorFlow models. Like the Kubeflow Pipelines SDK, TFX handles the Python-based authoring layer. You write your pipeline steps in Python. But TFX is more opinionated and comes with built-in components for common tasks like data ingestion, transformation, training, and evaluation.
An important distinction to keep straight for the exam: TFX is only the SDK layer. It defines your pipeline, but it does not orchestrate or execute it. TFX needs a separate runner to do that. The most common runner for TFX is Kubeflow Pipelines, but TFX also supports Apache Beam and Apache Airflow. And those runners still need infrastructure, whether that is Kubernetes, a local machine, or a VM.
When people say "TFX" or "TensorFlow Extended," they mean the TFX SDK. It is shorthand. The TFX SDK is analogous to the Kubeflow Pipelines SDK. Both are authoring layers. Neither one runs the pipeline on its own.
Vertex AI Pipelines replaces the orchestration layer and the infrastructure layer from the setups described above. It is a managed runner.
You still write your pipeline using either the Kubeflow Pipelines SDK or the TFX SDK. Nothing changes about the authoring layer. But instead of deploying your own Kubeflow Pipelines instance and managing Kubernetes infrastructure to execute it, you hand the pipeline off to Vertex AI Pipelines.
That means no Kubernetes cluster setup, no manual scaling, no separate orchestration engine to maintain. You focus on writing your pipeline logic, and Vertex AI takes care of executing it. Both SDKs compile down to a standard pipeline spec, and Vertex AI Pipelines handles the rest.
This is the key concept to understand for the PMLE exam: Vertex AI Pipelines is not a new way to write pipelines. It is a managed way to run them. The authoring layer (Kubeflow Pipelines SDK or TFX SDK) stays the same. The orchestration and infrastructure layers are what Vertex AI replaces.
Vertex AI Pipelines is worth understanding well for the Professional Machine Learning Engineer exam. The main areas to focus on are how DAGs structure pipeline workflows, why the acyclic property matters and how retraining still works within that constraint, the distinction between the authoring layer (Kubeflow Pipelines SDK or TFX SDK) and the orchestration layer, and the role Vertex AI Pipelines plays as a managed runner that eliminates the need to manage your own infrastructure. If those four things are clear, you should be well prepared for pipeline questions on the exam.
