Fix a failed CloudFormation stack

First find the earliest failed resource event: everything after it is usually fallout. Then the stack's status decides your only legal moves. Don't fix resources by hand unless a step below tells you to: changes outside CloudFormation are the most common reason a rollback can't finish.

1. Find the root event

describe-stack-events lists newest first. Read it oldest first and stop at the first *_FAILED. Its ResourceStatusReason holds the error from CloudFormation or the underlying service. (Reading for the first failure is a method, not a documented rule.) The console also groups events by operation ID and can filter to failures.

import { CloudFormationClient, paginateDescribeStackEvents, type StackEvent } from "@aws-sdk/client-cloudformation";

const cfn = new CloudFormationClient({});

/** Earliest failed event of the most recent operation, if any. */
export async function firstFailure(stackName: string): Promise<StackEvent | undefined> {
  const events: StackEvent[] = [];
  for await (const page of paginateDescribeStackEvents({ client: cfn }, { StackName: stackName })) {
    for (const e of page.StackEvents ?? []) {
      events.push(e);
      // Stop at the event that started the latest stack operation (newest-first order).
      if (e.ResourceType === "AWS::CloudFormation::Stack" && e.LogicalResourceId === stackName &&
          /_IN_PROGRESS$/.test(e.ResourceStatus ?? "") && e.ResourceStatusReason === "User Initiated") {
        return events.reverse().find((x) => x.ResourceStatus?.endsWith("_FAILED"));
      }
    }
  }
  return events.reverse().find((x) => x.ResourceStatus?.endsWith("_FAILED"));
}

const f = await firstFailure("my-stack");
console.log(f?.LogicalResourceId, f?.ResourceType, f?.ResourceStatusReason);

For EC2 bootstrap failures, also read /var/log/cloud-init.log and /var/log/cfn-init.log on the instance (Windows: C:\cfn\log). You can only do that if the instance survives, so retry with rollback disabled (step 2c). Match the reason text in reference/cloudformation/errors.md.

2. Act on the status

2a. ROLLBACK_COMPLETE (a create failed and was cleaned up)

Only delete is possible: the stack can't be updated. Delete it, fix the template, and create it again. The CDK reports this as "is in a failed state" (reference/cdk/errors.md).

2b. UPDATE_ROLLBACK_FAILED (an update failed, and so did the rollback)

Fix the cause, then continue the rollback to reach UPDATE_ROLLBACK_COMPLETE:

Cause in the event	Fix before continuing
Signals not received (Auto Scaling, wait conditions)	Send them: aws cloudformation signal-resource …
Resource changed or deleted outside CloudFormation	Recreate or restore it with the same name and properties as in the original template
Insufficient permissions	Add the missing modify actions (create alone isn't enough)
Invalid security token	Nothing. Continuing refreshes the credentials
Quota reached	Delete spare resources or raise the quota
Resource didn't stabilize	Nothing. Wait for the service, then continue

aws cloudformation continue-update-rollback --stack-name my-stack

If a resource still can't roll back, skip it:

aws cloudformation continue-update-rollback --stack-name my-stack \
  --resources-to-skip MyCustomResource ChildStackName.MyAsg

• You can skip only resources that went UPDATE_FAILED during the rollback. Skip as few as possible.
• Resources in a nested stack are named <NestedStackName>.<LogicalId>. A nested-stack resource itself can be skipped only when its stack is DELETE_IN_PROGRESS, DELETE_COMPLETE or DELETE_FAILED.
• --force-rollback skips every failed resource, nested ones included. You can't combine it with --resources-to-skip.
• Skipped resources are marked UPDATE_COMPLETE but no longer match the template, and drift detection shows them as NOT_CHECKED. Make the template and the resources agree before the next update, or the stack can become unrecoverable.

You can't create a change set on a stack in UPDATE_ROLLBACK_FAILED.

2c. CREATE_FAILED / UPDATE_FAILED with rollback disabled

This is the result of Preserve successfully provisioned resources: --disable-rollback on create-stack, update-stack or execute-change-set, or --on-failure DO_NOTHING on create. Resources that succeeded stay in place; on update, failed resources return to their last stable state. Your options:

• Retry when the fix is outside the template (an IAM permission, say):

aws cloudformation update-stack --stack-name my-stack --use-previous-template --disable-rollback

• Update with a fixed template. Keep --disable-rollback set, or the update is refused.
• Roll back: aws cloudformation rollback-stack --stack-name my-stack. This deletes the stack if it never had a stable state.

Updates that replace a resource can't be preserved this way.

Express mode disables rollback by default (--deployment-config '{"mode": "EXPRESS"}', cdk deploy --express). A failed express deployment therefore lands in *_FAILED rather than rolling back. Set "disableRollback": false if you want the old behaviour. The SDK's DeploymentConfig.DisableRollback field documents its own default as false; set it explicitly rather than rely on either default.

2d. DELETE_FAILED

1. Read the failed resource's reason. Typical causes are a non-empty S3 bucket, a security group still attached to instances or ENIs, or missing delete permissions on the underlying service.
2. Empty or detach it, then delete again. Or keep the resource and drop it from the stack:

aws cloudformation delete-stack --stack-name my-stack --retain-resources MyBucket

1. Or use --deletion-mode FORCE_DELETE_STACK (API DeletionMode), meant for stacks stuck in DELETE_FAILED. What it leaves behind isn't described on the fetched pages, so list the stack's resources first.

Termination protection blocks deletes and leaves the status unchanged. On nested stacks, it is the root stack's setting that counts. Delete nested stacks only through their root.

2e. Nested stacks stuck in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS, UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS

One nested stack failed to roll back, so CloudFormation halted cleanup for all of them. AWS's documented fix is to contact AWS Support. Out-of-band changes and short Auto Scaling signal timeouts are the named causes.

3. Before the next attempt

• Read reference/cloudformation/stack-operations.md for the timeout, service-role and capabilities settings that cause many of these failures.
• Preview the fix with a change set (reference/cloudformation/change-sets-drift-and-import.md).
• If someone edited resources by hand, run drift detection first.