AutoSxS vs. Reference-Based Metrics for Automated Model Evaluation

GCP Study Hub
November 5, 2026

Validating a candidate model against a baseline model already running in production requires evidence that the new version actually performs better before promotion. On Gemini Enterprise Agent Platform (formerly Vertex AI), Auto Side-by-Side (AutoSxS) automates that check by comparing responses from two models side-by-side on the same task instead of scoring outputs against a fixed reference text. For the Professional Machine Learning Engineer exam, evaluation scenarios frequently turn on recognizing when this pairwise comparison fits better than traditional reference-based scoring.

Why pairwise comparison differs from ROUGE and BLEU

Traditional evaluation metrics such as ROUGE and BLEU score a model's output against a reference text. Because they need a ground truth answer to measure overlap against, they run into limitations on open-ended tasks where many different answers are equally good.

AutoSxS skips the reference text entirely and evaluates two model responses directly against each other. Instead of measuring overlap with a single ground truth answer, it judges which of the two responses better satisfies the task criteria. For open-ended tasks where many valid responses exist, we would lean toward AutoSxS pairwise evaluation rather than reference-based metrics like ROUGE or BLEU.

How the AutoSxS evaluation workflow runs

AutoSxS works off predictions that both models have already generated on the same set of prompts. The end-to-end workflow follows a consistent sequence:

  • Model A and Model B each generate responses on the same input data.
  • Both sets of predictions are written to BigQuery or Cloud Storage, where AutoSxS expects to read them.
  • Agent Platform Pipelines orchestrates the comparison job end-to-end.
  • AutoSxS reads both prediction sets and evaluates them pair-by-pair using a large language model acting as a judge.
  • For each pair, the judge decides which response is better against the task criteria and returns a reason for the decision rather than just a verdict.
  • AutoSxS writes the individual preference judgments, explanations, and aggregate evaluation outputs back to BigQuery or Cloud Storage.

Returning a reason alongside each preference judgment lets teams sanity-check whether the automated judgments make sense rather than relying on bare verdicts. Writing both the per-pair judgments and the aggregate evaluation results back to BigQuery or Cloud Storage also keeps the outputs in a place teams can query and retain for the record.

Running AutoSxS in Agent Platform Pipelines

AutoSxS integrates directly with Agent Platform Pipelines, allowing the comparison to run as a pipeline job rather than something assembled by hand. Because it runs as a pipeline job, teams can schedule evaluations, rerun them whenever a new model version appears, and keep the results tracked alongside the rest of their workflow.

For the Professional Machine Learning Engineer exam, keep the architectural boundaries clear. AutoSxS reads pre-generated predictions from two models stored in BigQuery or Cloud Storage, uses a large language model judge to produce per-pair preference judgments with explanations and aggregate results without needing reference text, and writes those outputs back to BigQuery or Cloud Storage under the orchestration of Agent Platform Pipelines.

Our Professional Machine Learning Engineer course covers AutoSxS automated model evaluation alongside Agent Platform Pipelines and reference-based evaluation metrics, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow