BigQuery Logs for the PDE Exam: Cloud Logging Integration and Job IDs

BigQuery logging is one of those topics that sits quietly in the corner of the Professional Data Engineer exam guide and then shows up in a scenario question you did not see coming. The mechanics are not complicated, but the exam expects you to know which log type captures which kind of activity, and how to find a specific BigQuery job by ID inside Cloud Logging. I want to walk through what you actually need to retain for test day.

Every BigQuery activity flows into Cloud Logging

The first thing to internalize is that BigQuery does not run its own separate logging system. All activities and operations performed in BigQuery, whether by a human user running a query in the console or a service account kicking off a load job, are logged in Cloud Logging under the BigQuery resource type. That single fact resolves a surprising number of exam questions. If you see a scenario asking where to look for audit information about a BigQuery dataset modification, the answer is almost always Cloud Logging filtered by the BigQuery resource type.

Inside that resource type, BigQuery emits three distinct categories of logs, and the Professional Data Engineer exam likes to test whether you know the difference between them.

The three log types you need to know

The split between log types is not arbitrary. Each one answers a different question about your data warehouse.

• Admin Activity Logs capture configuration and management actions on BigQuery resources. Dataset creation, table deletion, schema changes, IAM policy updates on a dataset, anything that changes the shape or governance of your warehouse lands here. These are on by default and cannot be disabled.
• Data Access Logs record reads and writes against BigQuery tables. This is where you go when you need to know who queried a table, when a streaming insert hit a table, or which job loaded data into a particular partition. They are especially important for tracking specific data ingestion jobs, which is exactly the kind of phrasing the exam likes to drop into a question stem.
• System Event Logs capture BigQuery's automatic operations, things like internal resource allocation and maintenance the service performs on its own. You rarely need these for day-to-day debugging, but knowing they exist as a separate category is fair game.

A common exam pattern is to describe a situation (someone changed a dataset's permissions, a load job mysteriously wrote bad rows, a table started returning stale data) and ask which log type you would consult. Map the verb to the log: configuration verbs go to Admin Activity, read/write verbs go to Data Access, and anything BigQuery did to itself goes to System Event.

Finding a specific job by ID in Logs Explorer

The other piece of this topic that shows up on the Professional Data Engineer exam is the procedure for finding a specific BigQuery job in the logs. The workflow is short enough to commit to memory.

Start in the Cloud console and search for logging or cloud logging to land in Logs Explorer. At the top of the page is the query builder, which is the tool that translates your filter choices into a logging query.

From the Resources drop-down, select BigQuery as the resource type. The query builder updates to scope your search to BigQuery logs only. Then in the Log name field, choose cloudaudit.googleapis.com/activity to filter for audit logs related to BigQuery jobs. Again the query updates automatically.

At that point you can narrow further by job type or status. The two filter expressions worth memorizing are:

protoPayload.methodName="jobservice.insert"

which surfaces newly submitted jobs, and:

protoPayload.methodName="jobservice.jobcompleted"

which surfaces jobs that have finished, successfully or otherwise. If a question asks how you would track when a particular query started versus when it completed, those two method names are the answer.

Why job IDs matter

Every BigQuery job has a unique job ID, and that ID is your stable handle across the entire platform. The job ID appears in the BigQuery console job history, in the response when you submit a job via the API or the bq command line, and inside the protoPayload of the corresponding Cloud Logging entries. When you are debugging a failed load or trying to attribute cost to a particular query, the job ID is the join key.

If you already know the job ID, the fastest way to pull its log entry is to drop it into the Logs Explorer query as an additional filter on protoPayload.serviceData.jobCompletedEvent.job.jobName.jobId or simply search for the ID string in the free-text search box after you have scoped to BigQuery audit logs. The exam will not ask you to hand-write that field path, but it may ask you which tool you would use to look up a job's history after the fact, and the answer is Cloud Logging filtered to the BigQuery resource.

How this shows up on test day

Expect questions framed around audit and observability. Something like: a security team wants to know who modified the schema of a sensitive table. Or: an ingestion pipeline failed overnight and you need to find the exact failure reason for the failed load job. Or: you need to alert on any BigQuery job that exceeds a certain duration. In each case the path goes through Cloud Logging, the BigQuery resource type, and either Admin Activity or Data Access depending on whether the question is about configuration or data movement.

Knowing the three log categories, the cloudaudit.googleapis.com/activity log name, and the jobservice.insert and jobservice.jobcompleted method names is enough to handle almost every BigQuery logging question on the Professional Data Engineer exam.

My Professional Data Engineer course covers BigQuery logging end to end, including the Logs Explorer walkthroughs, the audit log field paths, and the exam-style questions that test whether you can map a scenario to the right log type.