Find the failing call in Cloud Logging, then alert on it
Six steps from "something is wrong" to a standing alert: query by resource and severity in the Logging query language, narrow to the verbatim error string, read the principal out of it, cross-check the trace, then turn the query into a log-based metric with an alerting policy — and a channel that has been tested.
markdownguidev19RNY-3QGHstage: operatetrust: officialfamily: observabilityservice: cloud-logginginterface: anyLogging query languageseverityresource.typelog-based metricalerting policynotification channeltrace correlationverbatim error string
Find the failing call
Step 1 — narrow by resource and severity, in that order
Query in the Logging query language, starting from the two filters that cut the most:
resource.type="<the service>"
severity>=ERROR
Resource first, because severity alone across a busy project returns everything that was ever unhappy.
Step 2 — narrow to the verbatim string
Take the error text exactly as the caller received it and search for it. Every error catalogue in this hub
is indexed that way, and so is a log query. Do not paraphrase and do not trim an error code
(reference/compute/errors.md, reference/storage/errors.md,
reference/data/errors.md, reference/identity/errors.md).
If nothing matches, ask whether the log is there at all before concluding the error did not happen:
reference/observability/errors.md §4 lists the three reasons it might be missing.
Step 3 — read the principal out of the error, not out of your assumptions
A Google Cloud permission error carries an error ID whose context is "the principal, resource, permission". In three common cases the principal is not the one you expect:
- Config Connector reconciles as its own service account, "not the identity of the human user or service
account that submitted the manifest";
- a Cloud Run deploy involves you, the runtime service account and the service agent;
- any CI pipeline acts as its own identity.
Diagnosing with the wrong principal answers the wrong question
(guides/identity/fix-permission-denied.md step 2).
Step 4 — cross-check the trace, knowing it may not exist
If the request was traced, the spans show where the latency or the failure sat. If there is no trace,
establish whether sampling means it was never recorded before you go looking for broken instrumentation —
those two look identical (reference/observability/cloud-trace.md §4).
Step 5 — turn the query into a log-based metric
A query you had to reconstruct is a query you will have to reconstruct again. Make the narrowed query a log-based metric; it then behaves like any other metric — chartable, alertable, usable in an SLO.
Step 6 — alert on it, and test the channel
- an alerting policy over the log-based metric;
- a notification channel attached;
- a test delivery through that channel.
Step 3 is the one that gets skipped, and a policy with a correct condition and an unverified channel fires
silently. That is worse than no alert, because the dashboard looks healthy
(reference/observability/errors.md §3).
Also confirm the resource's project is inside the metrics scope, or the policy watches a series that does not reach it.
For a migration
Do this before the migration, on the source platform, for the five errors that matter most. You then have
a baseline — the same five failures, counted — against which the target can be compared, and you discover the
observability gaps while there is still something to compare against. Alerting policies are re-created on
the far side rather than migrated, so the count of them is a direct effort estimate
(guides/parity/observability.md).