
A trained machine learning model can serve predictions in one of two deployment modes, batch inference or real-time inference, and the choice depends on business requirements and timing constraints. On Gemini Enterprise Agent Platform, real-time inference is handled by Agent Platform endpoints, which are also known as Agent Platform online prediction or Agent Platform online inference. For the Professional Cloud Architect exam, it helps to settle the deployment mode first and then reason about how endpoints receive models, split traffic, and should be placed.
Batch inference, also called offline inference, usually runs on a scheduled interval or is triggered by another process or by a person. It handles large volumes of data at low frequency, often processing many thousands or millions of records at once. The system is latency tolerant, meaning a run can take minutes or even hours to complete. The results are usually stored somewhere for later use, such as updating a database with customer risk scores.
Customer loan risk scoring is a common example. The entire customer base is processed overnight on a regular schedule to see who is at risk of not being able to pay back their loans. A batch server processes the large dataset on that schedule and writes the results back to a database, where other systems can read the pre-computed predictions when they need them.
Real-time inference, also called online inference, delivers millisecond-scale response times and processes prediction requests one after another in quick succession. Each prediction is returned immediately through an API response, following a low-volume but high-frequency pattern. Credit card fraud detection is a common example, since each transaction needs an instant decision while the customer is standing at the checkout. Requests come into a cloud API endpoint and predictions are returned immediately. There is no batch processing or storage step, just direct request and response cycles running continuously throughout the day.
The deciding factor between the two is the business need. If results can wait hours and everything needs to be processed at once, batch inference fits. If individual cases need instant decisions, real-time inference fits.
A trained model is not usable in its raw form. Training produces a file that contains the model's architecture, weights, and parameters, but that file cannot accept requests from users or applications on its own. To make a model usable, something has to load it into memory, accept HTTP requests, run inference on the input data, and return predictions, all while staying scalable and reliable.
Agent Platform endpoints provide that infrastructure. They are a managed service that takes care of the servers and containers needed to make a model useful in real applications.
Models can be deployed to Agent Platform endpoints from several sources: AutoML, BigQuery ML, Agent Platform custom training, and models trained elsewhere, a category that includes pre-trained models, models from Model Garden, and imported models. All of these sources first store the model in Model Registry, and the model is then deployed from the registry to an endpoint. In some specific cases a model can bypass Model Registry and go directly to an endpoint, but that path is not recommended.
Several models can be deployed to the same endpoint, and the endpoint can route incoming requests between them based on percentages you configure. Suppose model version one is the current production model that has been running reliably, and model version two is a newer version you want to test. With both deployed to the same endpoint, 75 percent of incoming traffic might go to version one and 25 percent to version two. Applications call one endpoint URL, and Agent Platform distributes the requests behind the scenes.
This setup has several practical benefits.
Traffic splitting is not limited to two models. A single endpoint can split traffic three ways, four ways, or more if needed. The endpoint handles the routing while applications keep using the same URL.
Location matters when deploying models to Agent Platform endpoints. When a model is deployed, the endpoint downloads the model files from Cloud Storage. If the model is stored in us-central1 but the endpoint is in europe-west1, that cross-region transfer adds latency to deployments and can slow down cold starts when new instances spin up. Storing models and endpoints in the same region gives faster deployments and better performance as the endpoint scales.
An Agent Platform endpoint works like any other HTTP endpoint, so other Google Cloud services can call it as well, depending on the architecture. Professional Cloud Architect scenarios may include such services alongside an endpoint, and the details of those services matter less than recognizing that nothing about an Agent Platform endpoint prevents other applications or Google Cloud services in the environment from calling it.
Our Professional Cloud Architect course covers Agent Platform deployment modes and online prediction endpoints alongside batch prediction and Model Registry, with practice questions that drill these distinctions.