Gen AI Evaluation Service: Rubric Metrics for Agents

GCP Study Hub
November 19, 2026

The Gen AI evaluation service provides adaptive rubric metrics, referenced in the SDK as types.RubricMetric.NAME, to grade specific aspects of how an agent behaves. Each metric isolates a different quality signal, ranging from whether claims in an answer are supported by tool results to how well the agent selects and orders tools across one or many turns. On the Professional Agentic Architect exam, picking the right evaluation approach depends on matching the metric to both the conversation shape and the behavior under review, and knowing when the Gen AI evaluation service fits better than local testing or operational monitoring.

Matching the metric to conversation shape and target

Selecting an adaptive rubric metric follows two steps. First, match single-turn or multi-turn to the shape of the evaluation cases. Second, choose the metric that grades the exact property you want to measure.

Keeping those two decisions separate prevents common measurement mismatches. A multi-turn metric is not designed for single-turn request and response cases, and a response quality metric cannot reveal whether an agent misused its tools on the way to an acceptable answer.

Response quality, instruction, and hallucination metrics

Four adaptive rubric metrics focus on the text of the agent's response rather than on how tools were invoked:

  • types.RubricMetric.GENERAL_QUALITY: grades overall response quality.
  • types.RubricMetric.FINAL_RESPONSE_QUALITY: evaluates the quality of the agent's final answer rather than the tool calls that led to it.
  • types.RubricMetric.INSTRUCTION_FOLLOWING: measures how well a response follows the constraints and instructions it was given, without evaluating tool invocation.
  • types.RubricMetric.HALLUCINATION: checks whether the claims made in a response are supported by the tool results.

These metrics serve distinct review goals. An agent can write a clear final answer that still fails to follow the constraints and instructions it was given, which INSTRUCTION_FOLLOWING detects, or that makes claims unsupported by what its tools returned, which HALLUCINATION detects. None of these four metrics inspects tool selection, parameter accuracy, or step ordering.

Evaluating tool use within a single turn

When each evaluation case is a single interaction and the goal is to audit how the agent interacts with tools, the appropriate metric is types.RubricMetric.TOOL_USE_QUALITY. This adaptive rubric metric evaluates three things within a single turn: selection of appropriate tools, correct parameter usage, and adherence to the expected sequence of operations.

An important property of TOOL_USE_QUALITY is that it does not take the final response as an input. That design isolates the tool-calling process from the final output. If an agent arrives at the right answer despite calling the wrong tool first, passing unnecessary parameters, or running steps in an inefficient order, response metrics like GENERAL_QUALITY and FINAL_RESPONSE_QUALITY will still score the answer well. TOOL_USE_QUALITY surfaces those tool-use defects directly.

Multi-turn rubric metrics

When conversations span multiple turns, single-turn metrics miss behavior that only emerges across the full exchange. For multi-turn conversations, these rubric metrics cover different aspects of the exchange:

  • types.RubricMetric.MULTI_TURN_TOOL_USE_QUALITY: evaluates tool selection and tool calls across every turn of a conversation. It catches sequencing and dependency errors that only appear across turns, such as invoking a dependent action before an earlier prerequisite call is confirmed.
  • types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY: evaluates the logic and efficiency of a conversation that spans many turns.
  • types.RubricMetric.MULTI_TURN_TASK_SUCCESS: measures whether the conversation reached its goal, not how the agent got there.

The distinction between MULTI_TURN_TASK_SUCCESS and MULTI_TURN_TOOL_USE_QUALITY is worth keeping clear. A multi-turn interaction can reach its goal even when the agent invoked tools in a wasteful or risky order. Measuring task success alone confirms the outcome but hides tool-calling mistakes along the path, while evaluating single-turn tool use in isolation misses how tool calls fit together across turns.

Gen AI evaluation service versus local ADK evaluation

Choosing an evaluation tool also depends on where tests run in the development lifecycle. Running the Gen AI evaluation service against an agent requires deploying the agent first, such as to staging or production. Local ADK evaluation does not require deployment and runs directly on a developer machine or inside CI.

When a team wants pre-deployment regression checks on both tool calls and final answers without deploying the agent, we would lean toward an ADK evalset run with adk eval, or through pytest with AgentEvaluator. The evalset records expected tool calls alongside reference responses, and a test_config.json file defines scoring criteria such as tool_trajectory_avg_score for tool calls and response_match_score for the final response.

Other local checks cover different layers and do not replace an evalset. Comparing trace spans against a baseline tracks timing and execution structure, not whether answers match a reference. Unit tests that invoke tool functions directly verify each function in isolation, so they cannot confirm whether the agent selects the right tools and passes the right arguments.

Tracking answer quality over time on deployed agents

After an agent is deployed, rubric metrics also support ongoing quality monitoring. Running the Gen AI evaluation service on a fixed schedule against a sample of production conversations, using response metrics such as GENERAL_QUALITY and FINAL_RESPONSE_QUALITY, produces comparable scores over time. Teams can use that trend to identify when answer quality began to drop after an upstream data change and to confirm whether subsequent fixes restore quality.

Standard infrastructure and machine learning signals do not capture this type of quality drift:

  • Latency, error rates, request volume, and error logs: valid responses that become vaguer or less helpful do not cause runtime failures, so operational dashboards and error logs stay unchanged.
  • Token counts: response length does not indicate whether an answer is useful.
  • Model Monitoring: detects feature skew and drift in inputs sent to prediction models, not the quality of an agent's answers.

For the Professional Agentic Architect exam, recognizing that less helpful responses represent a quality measurement problem rather than an error-rate or feature-skew problem points directly to scheduled sampling with Gen AI evaluation service rubric metrics.

Our Professional Agentic Architect practice exams test Gen AI evaluation service rubric metrics alongside local ADK evaluation and ongoing quality tracking, with detailed explanations that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow