Logs Insights queries for the hub's error catalogues
Copy-ready CloudWatch Logs Insights queries that find the evidence for the hub's error catalogues — Lambda timeouts, init and import errors, memory and latency from REPORT lines; VPC flow-log REJECTs by port and interface; CloudTrail AccessDenied by principal and who-deleted-what; API Gateway 4xx/5xx; Route 53 SERVFAIL; JSON application logs — each with the log group it runs against.
markdownexamplev4W5EK-WDA7stage: operatetrust: officialservice: cloudwatchinterface: consoleLogs Insights query examplesLambda Task timed out queryLambda REPORT duration memory queryRuntime.ImportModuleError querycold start queryVPC flow logs REJECT queryflow logs dstPortCloudTrail AccessDenied queryCloudTrail who deleted queryerrorCode statsAPI Gateway 5xx queryRoute 53 SERVFAILJSON log level ERROR querytop errors by countbin 5m
Run each query against the log group named in its heading, over the shortest window that contains the failure. Queries marked (AWS sample) are copied from AWS's examples page. The rest were composed for this hub from documented commands and fields and haven't been run against live logs, so adjust field names to your log format. Field and limit rules are in guides/cloudwatch/query-logs.md.
Lambda: /aws/lambda/<function>
Timeouts and runtime errors (strings from reference/lambda/errors.md):
fields @timestamp, @requestId, @message
| filter @message like /Task timed out after|Runtime\.[A-Za-z]+|ERROR/
| sort @timestamp desc
| limit 50
Count errors by kind, every 5 minutes:
filter @message like /Task timed out after|Runtime\./
| parse @message /(?<kind>Task timed out|Runtime\.[A-Za-z]+)/
| stats count(*) as n by kind, bin(5m)
| sort n desc
Slow invocations, one row per request (AWS sample). @duration is in milliseconds:
fields @timestamp, @requestId, @message, @logStream
| filter @type = "REPORT" and @duration > 1000
| sort @timestamp desc
| dedup @requestId
| limit 20
Latency over time (AWS sample):
filter @type = "REPORT" |
stats avg(@duration), max(@duration), min(@duration) by bin(5m)
Memory headroom (AWS sample; "provisonedMemoryMB" is AWS's spelling). A small overProvisionedMB means the function is close to running out of memory:
filter @type = "REPORT"
| stats max(@memorySize / 1000 / 1000) as provisonedMemoryMB,
min(@maxMemoryUsed / 1000 / 1000) as smallestMemoryRequestMB,
avg(@maxMemoryUsed / 1000 / 1000) as avgMemoryUsedMB,
max(@maxMemoryUsed / 1000 / 1000) as maxMemoryUsedMB,
provisonedMemoryMB - maxMemoryUsedMB as overProvisionedMB
JSON log format (LogFormat=JSON; application fields are timestamp, level, message and requestId):
fields @timestamp, level, message, requestId, errorType, errorMessage
| filter level = "ERROR"
| sort @timestamp desc
| limit 50
VPC Flow Logs: the flow log's log group (default format)
What got rejected, by destination port (reading the results: reference/vpc/flow-logs.md):
filter action = "REJECT"
| stats count(*) as rejects by dstAddr, dstPort, protocol
| sort rejects desc
| limit 25
Rejects for one instance or ENI (a timeout you're chasing):
fields @timestamp, srcAddr, srcPort, dstAddr, dstPort, action
| filter interfaceId = "eni-0123456789abcdef0" and action = "REJECT"
| sort @timestamp desc
Top talkers in a subnet (AWS sample):
filter isIpv4InSubnet(srcAddr, "192.0.2.0/24")
| stats sum(bytes) as bytesTransferred by dstAddr
| sort bytesTransferred desc
| limit 15
CloudTrail: a trail's log group
Failed calls by error (AWS sample, widened from one event to all):
filter ispresent(errorCode)
| stats count(*) by errorCode, errorMessage
Access denied, by who and what:
filter errorCode like /AccessDenied|UnauthorizedOperation/
| stats count(*) as denials by userIdentity.arn, eventSource, eventName
| sort denials desc
| limit 25
Then read a single errorMessage with guides/iam/fix-access-denied.md.
Who deleted things today:
fields @timestamp, eventName, eventSource, userIdentity.arn, sourceIPAddress
| filter readOnly = 0 and eventName like /^Delete/
| sort @timestamp desc
| limit 100
JSON booleans may compare as 0/1 or "false"; try readOnly = "false" if this returns nothing (UNVERIFIED type handling).
One resource's history:
fields @timestamp, eventName, userIdentity.arn, errorCode
| filter requestParameters.bucketName = "my-bucket"
| sort @timestamp desc
requestParameters field names depend on the API (bucketName, functionName, …). They are dropped from records over 100 KB.
API Gateway: access-log group (fields as in your access-log format)
Last 4xx (AWS sample):
fields @timestamp, status, ip, path, httpMethod
| filter status>=400 and status<=499
| sort @timestamp desc
| limit 10
5xx by path:
filter status >= 500
| stats count(*) as errors by path, status
| sort errors desc
Route 53 query logs
SERVFAIL by name (AWS sample):
filter responseCode="SERVFAIL" | stats count(*) by queryName
Any log group
25 most recent events (AWS sample):
fields @timestamp, @message | sort @timestamp desc | limit 25
Exceptions per hour (AWS sample):
filter @message like /Exception/
| stats count(*) as exceptionCount by bin(1h)
| sort exceptionCount desc