Observability error catalogue — the empty chart, the alert that never fires, the missing log, and the trace that was not sampled
Cause, check and fix for the failures where the signal itself is the thing that is broken. Each row separates "the system is fine and the observability is wrong" from the opposite, which is the only distinction that matters at three in the morning.
markdownreferencev102YH-299Hstage: operatetrust: officialfamily: observabilityservice: cloud-monitoringinterface: anyempty chartno dataalerting policy not firingmetrics scopemissing log entrylog retentionlog sinktrace not foundtrace samplinglabel cardinalitylog-based metric
Observability error catalogue
Every row answers one question: is the system broken, or is the observability broken?
1. The chart is empty
| cause | usually the monitored resource, not the metric. A time series is a metric type plus a resource, and a filter naming the wrong resource or label matches nothing |
| check | drop every filter and chart the bare metric type. If data appears, the filter is wrong. If not, go to §2 |
| fix | correct the resource type and labels. Read an actual label set from a known-good series rather than assuming the names |
2. The chart is empty and the metric genuinely has no data
| cause | the metrics scope does not include the project the resource lives in |
| check | which project is the scoping project, and is the resource's project inside the scope? |
| fix | add the project to the metrics scope. In a multi-project estate this is the single commonest cause, and it is a configuration you were supposed to design up front (reference/observability/cloud-monitoring.md §2) |
3. The alerting policy never fires
| cause | three candidates, in order of likelihood: the policy's filter matches no time series (§1); the condition's duration window is never satisfied; the notification channel is missing or unverified |
| check | does the policy's own condition preview show data? Is a channel attached, and does a test delivery arrive? |
| fix | fix the filter first. A policy with a correct condition and no channel fires silently, which is the worst of the three because the monitoring looks healthy |
4. A log entry is not there
| cause | three candidates: an exclusion discarded it; the sink routed it to a destination you are not looking at; or the bucket's retention passed and "log entries are stored for a specified length of time and are then deleted" |
| check | what sinks and exclusions apply to this log name? Which bucket should hold it, and what is that bucket's retention? |
| fix | whichever applies. Do not assume a default retention — the overview does not state one, and the figures live in a separate quotas document |
And one access-shaped variant: you may be reading through a log view that scopes you away from the entry. A log that "does not exist" for one principal and does for another is a view, not a retention problem.
5. A trace is missing
| cause | sampling. It may never have been recorded |
| check | what is the sampling configuration? |
| fix | nothing, if it was not sampled — and that is the point of this row. A trace absent because of sampling is indistinguishable from a trace absent because instrumentation is broken, so establish the sampling rate before you debug instrumentation. The overview names sampling as a topic and states no behaviour, so read that page |
6. The metrics bill grew and nothing was deployed
| cause | label cardinality. A label carrying a request id, a user id or a URL path creates a new time series per value |
| check | which custom or log-based metrics have high-cardinality labels? |
| fix | remove the label, or bucket it. Nobody chooses this; it arrives with a well-meaning userId label |
The logging equivalent is a high-volume, low-value log line with no exclusion. Both are the same lesson: in observability, volume is a design decision that is usually made accidentally.
Next: guides/observability/find-the-failing-call-in-cloud-logging.md,
guides/parity/observability.md.