Data Leakage for the Professional Machine Learning Engineer Exam

GCP Study Hub
September 8, 2026

Data leakage occurs when information that is unavailable at prediction time enters the training process and inflates evaluation performance. When a model is exposed to patterns from the validation or test data used to evaluate it, metrics appear unusually strong and create a false sense of confidence. The underlying issue often stays hidden until deployment, when the model encounters genuinely new production data and its performance drops significantly. For the Professional Machine Learning Engineer exam, recognizing how leakage enters a pipeline and understanding that it is a data quality issue rather than a modeling issue are essential distinctions.

The training-only rule for feature engineering

Standard machine learning workflows divide a dataset into three parts: training, validation, and test sets. That separation establishes clear boundaries for how information flows. Any statistics or parameters calculated for feature engineering must come exclusively from the training data, and only afterward be applied to the validation and test sets. This rule applies across summary statistics, normalization parameters, encoding mappings, and imputation values.

Consider a model that uses a feature called average purchase amount by user. After splitting the dataset, you calculate each user's average purchase amount using only the training split. User A might average $50 and user B might average $30 across the training records. When transforming the validation and test sets, any user who appeared in the training data keeps that training-based average. If user A appears in the validation or test set, the value stays $50 rather than a new average calculated from those splits. For users who never appeared in the training data, you assign a default value derived from training, such as the overall average across all training users.

This boundary mirrors what happens in production. When a new customer visits a website, the system does not know their future purchase history and does not have statistics that include their behavior. Computing feature averages from data that includes future customers leaks information that will not exist at prediction time.

Three common sources of data leakage

Data leakage is fundamentally a data quality problem, much like class imbalance, rather than an issue with model architecture or hyperparameter tuning. Even minor leakage can make a model appear highly accurate during evaluation while performing poorly in deployment. Treating leakage prevention as a routine part of data preparation makes it easier to catch early across three main sources:

  • Features derived from the target variable: Including features that reflect what the model is trying to predict creates a circular dependency, even when the connection is indirect. Predicting patient recovery while including a treatment effectiveness rating assigned after observing recovery leaks the outcome into the inputs. Similarly, predicting loan defaults with a loan default risk score calculated using knowledge of actual defaults gives the model the answer ahead of time.
  • Global statistics calculated before splitting: Running preprocessing across the entire dataset before separating training, validation, and test sets is one of the most common sources of leakage. Scaling transaction amounts with the minimum and maximum across all transactions, or imputing missing ages with the median age of the full dataset, incorporates validation and test information directly into training features.
  • Improper handling of time series data: Sequential data depends on temporal ordering, and any feature that looks forward in time introduces leakage. Predicting this week's stock prices using a rolling average that includes next week's prices during training, or training on doctor's notes written after the medical outcome being predicted, feeds future knowledge into the model.

Preventing leakage in time series workflows

Time series problems are especially prone to leakage because sequential patterns make it easy to include future data accidentally. Scenarios on the Professional Machine Learning Engineer exam frequently focus on two distinct ways time series leakage happens: improper data splitting and features that draw on future values.

In a valid time series split, records remain in chronological order so past data always predicts future outcomes. For example, a full year of records can be divided with January through July 2024 in the training set, August through October in the validation set, and November and December in the test set. By contrast, scattering training, validation, and test samples randomly across the timeline causes leakage. A randomized split can end up using September data to predict June outcomes or November data to predict August outcomes, violating the temporal nature of the problem and producing unrealistic evaluation results.

Even with a chronological split, feature engineering can introduce future information:

  • Aggregated statistics that incorporate future periods, such as a six-month average that includes months after the prediction target.
  • Using day t+1 observations to predict outcomes on day t before those observations are available.
  • Normalizing values across all time periods instead of calculating normalization parameters strictly from historical training data.
  • Using whether a user converted within seven days to predict an action happening today, since that conversion outcome is future knowledge unavailable at prediction time.

Maintaining strict chronological boundaries in both data splits and feature engineering is necessary whenever predictions depend on time.

Recognizing data leakage in performance metrics

Data leakage also leaves recognizable signals in evaluation metrics. The main symptom is extremely high performance right from the start of experimentation. Achieving 99% AUC in the first few experiments without sophisticated modeling or hyperparameter tuning is a strong warning sign that target information may have entered the training features.

Comparing learning curves with AUC on the vertical axis and the number of epochs on the horizontal axis illustrates this difference clearly. In a normal learning curve, both training and validation AUC begin around 55% and climb steadily over 50 epochs, with training reaching roughly 82% and validation settling around 78%. That gradual improvement and small gap between training and validation reflect realistic learning. In a suspicious learning curve pointing to data leakage, training AUC jumps to 99% by the second or third epoch and remains there. Realistic training does not reach near-perfect performance almost immediately, so that behavior signals that the model is likely accessing information it should not have.

Why model changes do not fix data leakage

Changing the model, reducing model complexity, or tuning hyperparameters will not resolve data leakage, because architectural adjustments cannot remove leaked information the model should never have accessed in the first place. On the Professional Machine Learning Engineer exam, we would examine data splits, preprocessing boundaries, and feature availability at prediction time before considering any model-level adjustments.

Our Professional Machine Learning Engineer course covers data leakage alongside class imbalance and model evaluation metrics, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow