Cloud Build for ML: Retraining Pipelines and Cross-Project Deployment

GCP Study Hub
October 25, 2026

Cloud Build handles continuous integration and continuous delivery for machine learning workflows on Google Cloud by connecting version-controlled code changes to automated testing, retraining, and deployment. For the Professional Machine Learning Engineer exam, two Cloud Build patterns are worth understanding clearly: triggering model retraining pipelines on Gemini Enterprise Agent Platform (formerly Vertex AI) when repository code changes, and promoting tested assets across separate development and production projects. In both patterns, automated tests run before changes trigger downstream training or move into a production environment.

Code-triggered CI/CD for model retraining

In a code-triggered retraining workflow, an engineer iterates on a model by adjusting how features are calculated or by testing a different neural network architecture. Once those edits are complete, the engineer commits the changes and pushes them to a Git repository that stores both the model code and the feature engineering logic.

Pushing to the repository automatically triggers a Cloud Build job. Cloud Build prepares the execution environment and checks the code before any model training starts:

  • It installs the required dependencies, such as TensorFlow or scikit-learn, along with the Agent Platform SDK for pipelines and related utilities.
  • It runs a suite of tests, depending on the environment where the workflow is executing, to confirm that the changes have not broken existing functionality.
  • If those tests pass, Cloud Build triggers the machine learning pipeline in Agent Platform.

Model retraining inside Agent Platform

Cloud Build does not perform the retraining itself. Instead, it hands off execution to the Agent Platform pipeline, which runs the end-to-end training and validation workflow:

  • Pulls the latest training data from BigQuery.
  • Applies the updated feature engineering logic committed by the engineer.
  • Trains a new model using the updated architecture.
  • Validates the trained model's performance against predetermined benchmarks.
  • Deploys the model to serving endpoints if it meets the required quality thresholds.

When deployment completes, a freshly trained model is live and ready to handle prediction requests, all triggered automatically by a code push. This design offers three practical advantages. First, training only runs when code changes, which avoids unnecessary compute costs. Second, Cloud Build manages dependency installation and environment setup, eliminating a common source of deployment problems. Third, the workflow maintains a clear separation between data science development and infrastructure management.

Cross-project deployments with Cloud Build

While a full CI/CD lifecycle can span four separate environments (development, testing, staging, and production), not every architecture needs all four. In some cases, including scenarios on the Professional Machine Learning Engineer exam, teams use a simpler two-environment setup with just a dev project and a prod project.

There are two ways to structure Cloud Build across those projects:

  • Single Cloud Build instance in dev: Cloud Build is configured inside the dev project. When a change is ready, it deploys to the dev environment first so the change can be tested and verified. Once tests pass, that same Cloud Build instance promotes the deployment to the prod environment while still running from the dev project.
  • Separate Cloud Build instances per project: Each project maintains its own Cloud Build configuration and deploys independently within its own environment.

Both approaches are valid and carry different trade-offs. When a single Cloud Build instance in the dev project manages the full path from dev to prod, one permission requirement must be in place: the Cloud Build service account in the dev project needs access to deploy into the prod project. Without that cross-project access, promotion to the upper environment will fail even after tests pass in dev.

Promoting Managed Service for Apache Airflow DAGs from dev to prod

A common cross-project scenario involves deploying DAG files to Managed Service for Apache Airflow (formerly Cloud Composer) in production only after they pass tests in development. With a dev project and a prod project, Cloud Build in the dev project manages both initial testing and production promotion:

  • Cloud Build starts in the dev project and copies the updated DAG files to a Cloud Storage bucket in dev.
  • The dev Managed Service for Apache Airflow environment reads the DAG files directly from that dev Cloud Storage bucket, allowing the workflows to be tested and verified.
  • Only after tests succeed, Cloud Build copies the validated DAG files from the dev Cloud Storage bucket to the prod project's Cloud Storage bucket.
  • The prod Managed Service for Apache Airflow environment reads directly from its own Cloud Storage bucket and immediately begins using the promoted DAG files.

By gating the copy step behind passing tests in dev, Cloud Build ensures that only verified DAG files reach production. That structure protects workflow stability and reliability while keeping both the initial dev deployment and the final promotion to prod inside one automated process.

Our Professional Machine Learning Engineer course covers Cloud Build CI/CD workflows alongside Agent Platform pipelines and Managed Service for Apache Airflow orchestration, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow