Training-Serving Skew and Data Drift: Monitoring Models on Google Cloud

GCP Study Hub
October 10, 2026

In Gemini Enterprise Agent Platform (formerly Vertex AI), training-serving skew and data drift describe related but distinct monitoring conditions. Skew refers to any mismatch between the distribution of training data and the distribution of serving data in production, whether that gap appears on day one or develops later. Data drift refers to changes in production data over time that gradually degrade model performance. The Professional Machine Learning Engineer exam tests both the boundary between skew and drift and the specific subtypes of drift that appear in production.

How training-serving skew differs from data drift

In general machine learning, skew can carry different meanings depending on context. On Google Cloud and in Agent Platform, skew specifically means a difference between the training distribution and the serving distribution. That is why Agent Platform model monitoring provides skew detection as its own option, separate from data drift monitoring.

  • Skew from data drift: a model begins serving data that matches its training distribution, and over time the serving distribution shifts away. That change over time is data drift, which also creates skew relative to the training data.
  • Skew from day one: a model is trained on one dataset, deployed, and immediately receives serving inputs from a different distribution. No gradual drift over time has occurred yet, yet training-serving skew is already present.

Skew can result from data drift, but the two are not the same. Data drift tracks change over time, while skew captures any gap between training data and serving data regardless of when or how it occurs.

Why data drift degrades production models

Production data rarely stays static, and shifts often unfold gradually enough to miss until prediction quality drops. Even a well-built model produces worse predictions when the live environment diverges from its training environment.

As an illustrative scenario, consider user behavior before and after mobile adoption. Before 2012, about 80% of users might have been on desktop, averaging 10-minute sessions mostly on weekdays. After 2016, mobile might reach 60% of traffic, with 30-second average sessions spread across all hours and days of the week. Even with no change to the model's design, serving it without adjustment degrades predictions because the world it learned no longer matches the world it sees.

Feature drift

Feature drift is a change in the statistical distribution of input features over time. It can occur even when the output distribution and the relationship between inputs and outputs stay the same.

In an e-commerce fraud example, suppose pre-2012 traffic was 80% desktop ($85 average purchase) and 20% mobile ($25 average purchase), with an overall fraud rate of 2.1%. After 2016, desktop still averages $85, mobile still averages $25, and the overall fraud rate remains 2.1%, but the traffic mix flips to 60% mobile and 40% desktop. Fraudulent behavior itself has not changed, and no single input became anomalous, yet the proportion of inputs shifted. Because the model trained on comparatively little mobile data, it may not generalize well once mobile dominates.

Target drift

Target drift is a change in the statistical distribution of the target variable the model predicts, even while input features and feature-to-label relationships remain intact.

Suppose an industry data breach exposes email addresses, passwords, or payment details, prompting attackers to test accounts, make small purchases, and exploit vulnerabilities. Before the breach, the fraud rate is 2.1%, and the model's alert threshold is calibrated to that baseline using signals like unusual transaction amounts for a device type, suspicious login locations, and rapid bursts of user activity. After the breach, the fraud rate climbs to 6%. The model still applies the same decision logic to the same input features, but its baseline threshold now underestimates risk, allowing high-probability fraud cases to score too low and pass undetected.

Concept drift

Concept drift is a change in the real-world relationship between input features and the output label, and it is often harder to detect because input and output distributions can look unchanged.

Consider a fraud model using login distance from a user's typical location and transaction speed in transactions per hour. Initially, almost all fraud occurs when login distance exceeds 1,200 miles, so the model learns that distant logins indicate fraud. Later, fraudsters adapt. Transaction volume, the spread of login distances, and the distribution of transaction speeds all stay identical (zero feature drift), and the fraud rate remains 30% fraud and 70% non-fraud (zero target drift). What changed is the boundary: fraud now clusters at speeds above 70 transactions per hour, invalidating the rules the model learned.

In practice, concept drift often occurs alongside feature drift or target drift, and feature drift can directly cause concept drift. When desktop accounted for 80% of traffic and mobile purchases averaged $25, a $200 mobile purchase stood out as suspicious. Once mobile grew to 60% of traffic and average mobile purchases rose to $65, a $200 mobile purchase became normal even though the overall fraud rate remained 2.1%. The shifted input distribution made the old decision rule unreliable without any change in the target rate.

Feature attribution drift

Feature attribution drift tracks changes in how much each feature actually contributes to predictions in production. It is commonly measured with SHAP values, which show how much each feature moves an individual prediction before those values are aggregated across predictions.

For example, a model might initially rely more on login distance (higher SHAP value) than on transaction speed, then flip to leaning more heavily on transaction speed even though neither the model nor the input data has changed. On the Professional Machine Learning Engineer exam, keep two boundaries clear:

  • Feature attribution versus feature importance: feature importance measures how much a model structurally relies on a feature (what it was built to pay attention to), whereas feature attribution measures what it is actually paying attention to across live predictions.
  • Feature attribution drift versus concept drift: concept drift reflects an external shift in the real-world relationship between inputs and outputs. Feature attribution drift requires no external shift and only shows that the features contributing most to calculated predictions have changed.

Choosing the right signal in Agent Platform monitoring

When diagnosing production issues, we separate these signals by baseline and mechanism. Check training-serving skew for any gap between training data and live serving inputs, whether on day one or later. Check feature drift when input distributions shift over time, target drift when the label distribution shifts, concept drift when the input-to-label relationship changes, and feature attribution drift when aggregated SHAP contributions change in production.

Our Professional Machine Learning Engineer course covers training-serving skew and data drift alongside Agent Platform model monitoring and explainability, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow