Online vs. Batch Prediction on Agent Platform: When to Use Each

GCP Study Hub
October 30, 2026

Gemini Enterprise Agent Platform (formerly Vertex AI) supports two primary model deployment modes: online prediction, also known as real-time prediction, and batch prediction, also called offline prediction. Choosing between them depends on data volume, request frequency, latency tolerance, and how downstream systems consume results. On the Professional Machine Learning Engineer exam, architecture scenarios frequently describe a workload's timing and scale requirements and expect you to match them to the right serving mode.

Comparing online and batch prediction

Online prediction is built for millisecond-scale responses. Requests typically arrive one after another, and the model's API returns immediate responses. At any given moment, the model may only be computing a handful of predictions, which makes the access pattern low volume and high frequency. Credit card fraud detection is a standard example of an online workload. The application needs a prediction as soon as a transaction occurs rather than waiting a week, after fraudulent activity would already have taken place.

Batch prediction handles data that is large in volume and low in frequency. Jobs are usually scheduled or triggered rather than responding to individual incoming requests. Batch workloads are latency tolerant, meaning a job can run for minutes or hours without disrupting the business process. Instead of returning immediate API responses, batch prediction stores its results somewhere for later use, such as updating a database. Weekly customer loan risk scoring fits this pattern well. A lender can run a model once a week to score the risk of customers not being able to repay their loans instead of scoring continuously when real-time updates are not needed.

  • Online (real-time) prediction: Low volume, high frequency, millisecond-scale latency, one request after another, immediate API responses.
  • Batch (offline) prediction: High volume, low frequency, latency tolerant (minutes or hours), scheduled or triggered runs, results written to storage or a database for later use.

How Agent Platform batch prediction works

Batch prediction on Agent Platform is designed to process large datasets in bulk for high-volume, non-real-time prediction tasks. You point a batch prediction job at your input data, Agent Platform runs that data through your model, and the job writes the completed predictions to a destination you choose.

Agent Platform batch prediction accepts input data from three sources:

  • BigQuery tables
  • JSONL files stored in Cloud Storage
  • A JSONL file uploaded directly in the Agent Platform batch prediction interface

Once the job finishes processing the input records through the model, it writes the output to one of two destinations:

  • BigQuery tables, where teams can run further analysis on the predictions
  • JSONL files stored in Cloud Storage

Common batch prediction use cases

Batch prediction fits periodic workloads where a business evaluates a large collection of records at once rather than scoring single events as they happen. Common examples on Agent Platform include:

  • Customer churn prediction: Analyzing an entire customer base monthly or quarterly to identify which customers are likely to leave, so teams can reach out proactively with retention offers.
  • Demand forecasting: Running weekly or monthly predictions across thousands of products to optimize inventory levels and supply chain planning.
  • Lead scoring: Using a CRM database as input to score and rank thousands of sales leads, giving a sales team a prioritized list for outreach campaigns.

Each of these scenarios involves a large dataset that does not need to be evaluated in real time, making bulk processing on a schedule a natural fit.

Cost and deployment considerations

A key operational advantage of Agent Platform batch prediction is that it does not require a deployed endpoint. Because you do not need to keep an endpoint deployed, you only pay for the compute used during the batch job itself. For periodic, large-scale jobs, paying only for execution compute makes batch prediction a cost-effective option.

When evaluating serving architectures for the Professional Machine Learning Engineer exam, we would lean toward batch prediction whenever a workload processes large datasets on a schedule or trigger, tolerates minutes or hours of latency, and writes outputs to BigQuery or Cloud Storage for downstream use. When an application requires immediate, millisecond-scale API responses for high-frequency individual requests such as fraud detection, online prediction is the right choice.

Our Professional Machine Learning Engineer course covers online and batch prediction on Agent Platform alongside model deployment and serving workflows, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow