Serving Real-Time Models with Agent Platform Endpoints

GCP Study Hub
October 15, 2026

In its raw form, a trained machine learning model is a file containing architecture weights and parameters, and it cannot accept requests from users or applications on its own. Serving real-time predictions requires infrastructure that loads the model into memory, accepts HTTP requests, runs inference on input data, and returns predictions reliably at scale. Agent Platform Endpoints, also known as Agent Platform online prediction or online inference on Gemini Enterprise Agent Platform (formerly Vertex AI), provide this managed serving infrastructure by handling the underlying servers and containers. The Professional Machine Learning Engineer exam tests how to configure these endpoints across networking options, deployment patterns, regional placement, and autoscaling.

Deploying models through Model Registry

Models can reach Agent Platform Endpoints from AutoML, BigQuery ML, Agent Platform custom training, or other sources such as pre-trained models, models from Model Garden, and imported models trained elsewhere. Across all of these sources, the standard workflow is to store the model in Model Registry first and deploy it to an endpoint from there. Certain cases allow bypassing Model Registry to deploy directly to an endpoint, though we generally recommend keeping Model Registry in the path.

Public and private endpoint types

Agent Platform Endpoints fall into two broad categories: public endpoints reachable over the public internet and private endpoints that keep traffic inside your own network. The four options differ in networking architecture, latency predictability, security, and per-request payload size limits.

  • Dedicated public endpoint: Uses dedicated networking infrastructure over the public internet and is the default recommended option for general use. Because networking resources are not shared with other traffic, latency is more predictable and requests support a 10 megabyte payload limit.
  • Shared public endpoint: Uses shared networking infrastructure without dedicated resources, giving the default experience with a tighter 1.5 megabyte payload limit.
  • Dedicated private endpoint: Routes requests and responses privately inside Google's network, removing a network hop by avoiding the public internet. Recommended for production enterprise applications needing lower latency and stronger security, with a 10 megabyte payload limit.
  • Private endpoint: Routes traffic privately through VPC Network Peering between your VPC and the serving network. Also recommended for production enterprise applications requiring lower latency and stronger security, with a 10 megabyte payload limit.

If requests exceed the payload size limit, you need to send less data per call, which typically means fewer instances per request and more requests overall.

Deployment patterns: traffic splitting and multiple endpoints

Two deployment patterns appear frequently on the Professional Machine Learning Engineer exam, and each addresses a different operational need.

The first pattern places multiple models behind the same endpoint. More than one model lives behind a single serving address, so client applications keep calling the same URL with no code changes. Behind the endpoint, traffic splitting routes requests according to percentages assigned to each deployed model. For example, an endpoint can send 75% of live traffic to a stable production model (version 1) and 25% to model version 2 acting as a canary, a new version given a small share of live traffic to observe its behavior before committing to it. Because adjusting percentages is an endpoint configuration change rather than a code change or redeployment, this pattern supports three workflows:

  • Gradual rollout: Increase a new version's traffic share over time, such as moving from 5% to 25%, then 50%, and eventually 100% once trusted.
  • Risk mitigation: Limit user exposure if the new version returns bad predictions or latency spikes, and route traffic back to the stable version without redeploying.
  • A/B testing: Serve both versions on live production traffic at the same time to compare accuracy, latency, and business metrics rather than relying on offline evaluation alone.

The second pattern deploys models to multiple endpoints, giving each deployment its own serving address and separate infrastructure. The same model artifact, such as version 1, can run on a low-performance endpoint with smaller machines and a low replica count alongside a high-performance endpoint with stronger machines, possibly accelerators, and a higher replica count. This lets you assign appropriate resources to different environments (such as development or a low-traffic internal tool versus a customer-facing production app), isolate demanding workloads so a traffic burst on one application does not affect another's latency, and avoid over-provisioning low-demand apps that would otherwise share capacity sized for the heaviest consumer. The trade-off is cost: a model deployed to two endpoints runs as two independent sets of nodes, so you pay for both deployments separately in exchange for isolation and independent scaling.

Regional placement for deployment and inference latency

Regional placement governs two latency paths. First, store model files (such as artifacts in Cloud Storage) and the Agent Platform endpoint in the same region to lower deployment latency. During deployment, the endpoint downloads model files from Cloud Storage, so placing a model in us-central1 while running the endpoint in europe-west1 adds cross-region transfer latency and can slow cold starts when new instances spin up. Second, deploy endpoints in the region closest to your users to lower inference latency, since every prediction request travels from the user to the endpoint and back.

Autoscaling, worker utilization, and scaling metrics

Endpoints autoscaling automatically adjusts the number of inference nodes based on real-time traffic, adding nodes as requests increase and removing them as demand drops. Min-replica count and max-replica count define the lower and upper node limits. Setting min-replica count is a balancing act: keeping it close to expected baseline traffic reduces scale-up delay during sudden spikes, while setting it too high wastes cost during quiet periods. Scale-up takes time because a new node must be provisioned and load the model before serving, leaving existing nodes to absorb traffic while latency climbs. Warm nodes let spikes land on capacity that is already serving.

Each node runs a container that serves the model through workers, which are processes handling prediction requests. If only one worker runs on a multi-core machine, most of the machine sits unused regardless of request volume. CPU usage provides the signal: if CPU stays low during high traffic, the node likely has too few workers. Increasing the worker count lets the node use its resources fully and scale properly. Because autoscaling reacts to resource utilization, queuing requests paired with low CPU usage make a node look underused, so the autoscaler never triggers a scale-up even while users wait.

By default, autoscaling targets 60% CPU utilization. When GPUs are attached, it tracks whichever utilization is higher between CPU and GPU against the same 60% target, so an accelerator-bottlenecked model still scales even if the CPU looks idle. You can override these defaults with the autoscaling metric specs configuration:

  • CPU utilization: Set CPU utilization with a custom target. When configured as the only scaling metric, GPU usage will not trigger scaling at all.
  • GPU duty cycle: Scale directly on GPU duty cycle, the fraction of time the GPU is actively processing, when the accelerator is doing the real work and CPU is not the bottleneck.

Calling endpoints from other Google Cloud services

Because an Agent Platform endpoint works like any standard HTTP endpoint, other applications and Google Cloud services can invoke it directly depending on your architecture. On the Professional Machine Learning Engineer exam, scenarios often pair Agent Platform Endpoints with other Google Cloud services, and there are no special restrictions preventing those services or other applications from calling an endpoint over HTTP.

Our Professional Machine Learning Engineer course covers Agent Platform Endpoints alongside Model Registry and Agent Platform custom training, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow