Create CloudWatch alarms that fire when they should

Alarm on a rate or a symptom users feel, require M breaching points out of N, and decide on purpose what "no data" means. The default, TreatMissingData: missing, sends a quiet metric to INSUFFICIENT_DATA, which pages nobody. An error-count metric that emits nothing on good days needs notBreaching.

The settings that decide behaviour

How missing data is handled:

• CloudWatch looks a little further back than N periods with the default sliding windows.
• If it finds at least N real points, the missing-data setting isn't used. Otherwise it fills the gaps with the setting, as few times as possible.
• missing goes to INSUFFICIENT_DATA only when every point in the range is missing.
• AWS/DynamoDB alarms default to ignore.

The evaluation window (period × N) can be at most 7 days for periods of an hour or more, and 1 day for shorter periods.

Actions run only when the state changes. An alarm that stays in ALARM doesn't notify again; Auto Scaling actions are the exception and repeat every minute. CloudWatch doesn't check that the SNS topic or other target exists, so test it. Metric alarms can notify SNS, act on EC2 or Auto Scaling, open Systems Manager OpsItems or incidents, and start CloudWatch investigations.

Error rate for a Lambda function (TypeScript SDK)

Errors counts function errors, timeouts included, but not throttles. Invocations excludes throttled requests. So the rate below is "share of runs that failed". Alarm on Throttles separately.

import { CloudWatchClient, PutMetricAlarmCommand } from "@aws-sdk/client-cloudwatch";

const cw = new CloudWatchClient({});
const fn = "orders-handler";
const dims = [{ Name: "FunctionName", Value: fn }];

await cw.send(new PutMetricAlarmCommand({
  AlarmName: `${fn}-error-rate`,
  AlarmDescription: "More than 5% of invocations failed in 3 of 5 minutes",
  ComparisonOperator: "GreaterThanThreshold",
  Threshold: 5,
  EvaluationPeriods: 5,
  DatapointsToAlarm: 3,
  TreatMissingData: "notBreaching",            // no traffic is not an outage
  AlarmActions: ["arn:aws:sns:<region>:<account-id>:oncall"],
  OKActions: ["arn:aws:sns:<region>:<account-id>:oncall"],
  Metrics: [
    { Id: "errors", ReturnData: false,
      MetricStat: { Metric: { Namespace: "AWS/Lambda", MetricName: "Errors", Dimensions: dims }, Period: 60, Stat: "Sum" } },
    { Id: "invocations", ReturnData: false,
      MetricStat: { Metric: { Namespace: "AWS/Lambda", MetricName: "Invocations", Dimensions: dims }, Period: 60, Stat: "Sum" } },
    { Id: "rate", Label: "Error %", ReturnData: true,
      Expression: "100 * FILL(errors, 0) / invocations" },
  ],
}));

• In metric math, missing values count as 0 in arithmetic, and a point that would divide by zero is dropped. So minutes with no invocations produce no point, and notBreaching covers them.
• Exactly one entry has ReturnData: true: that's the series the alarm evaluates.
• IDs start with a lowercase letter, and function names are uppercase.
• Avoid scalar functions such as SUM(m1) on a single series in alarms.

The same alarm in the CDK:

import { Duration } from "aws-cdk-lib";
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch";
import type * as lambda from "aws-cdk-lib/aws-lambda";
import type { Construct } from "constructs";

export function errorRateAlarm(scope: Construct, fn: lambda.IFunction) {
  const rate = new cloudwatch.MathExpression({
    expression: "100 * FILL(errors, 0) / invocations",
    usingMetrics: {
      errors: fn.metricErrors({ statistic: "Sum", period: Duration.minutes(1) }),
      invocations: fn.metricInvocations({ statistic: "Sum", period: Duration.minutes(1) }),
    },
    label: "Error %",
  });
  return rate.createAlarm(scope, "ErrorRate", {
    threshold: 5,
    evaluationPeriods: 5,
    datapointsToAlarm: 3,
    comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
    treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
  });
}

Attach a notification with alarm.addAlarmAction(new SnsAction(topic)) from aws-cdk-lib/aws-cloudwatch-actions.

Other alarm types

Alarm stuck or noisy?