Amazon ECS with a deployment circuit breaker support detection of failed rollouts and can automatically roll back to the last successful deployment. Containers can fail to start or serve errors. The deployment needs to detect these failures and recover.
Amazon ECS provides a deployment circuit breaker for this purpose. For ECS rolling deployments, it can detect when a deployment fails to stabilize, stop further attempts, and roll back to a previous successful deployment. AWS documentation
This post walks through that mechanism using a small Terraform project: a Flask application running on ECS Fargate, an ECR repository for its images, and a deliberately broken release.
The deployment we want to protect
The application is intentionally simple. It exposes an HTTP endpoint on port 5000 and a health endpoint:
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
return "Hello from ECS!"
@app.route("/health")
def health():
return "OK", 200Three steps:
- Deploy a working version and let it complete.
- Deploy a broken version.
- Observe ECS detect the failure and restore the working version.
Establishing that first successful deployment matters: ECS needs a deployment in the COMPLETED state to use as a rollback target. AWS documentation
Deployment controller and circuit breaker
The deployment controller determines who manages the rollout, while the circuit breaker detects deployment failures and optionally triggers a rollback. With the ECS controller, ECS handles task replacement throughout the rollout.
We use the default ECS deployment controller because the deployment circuit breaker is supported only for ECS rolling deployments; it is not available with the CODE_DEPLOY or EXTERNAL controllers.
The Terraform configuration looks like this:
resource "aws_ecs_service" "app" {
name = "rollback-demo"
cluster = aws_ecs_cluster.app.id
task_definition = aws_ecs_task_definition.app.arn
desired_count = 2
launch_type = "FARGATE"
deployment_controller {
type = "ECS"
}
deployment_circuit_breaker {
enable = true
rollback = true
}
network_configuration {
subnets = var.subnet_ids
security_groups = [aws_security_group.app.id]
}
}This is an excerpt; it assumes the referenced resources and suitable network connectivity already exist.
To send container logs to CloudWatch, the ECS task execution role needs permission to create log streams and publish log events. It also needs permission to pull the container image from ECR. The following configuration assumes the ECR repository and CloudWatch log group are created separately.
data "aws_iam_policy_document" "ecs_execution_policy" {
statement {
sid = "PullContainerImage"
actions = [
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
]
resources = [aws_ecr_repository.app.arn]
}
statement {
sid = "AuthenticateToECR"
actions = ["ecr:GetAuthorizationToken"]
resources = ["*"]
}
statement {
sid = "WriteContainerLogs"
actions = [
"logs:CreateLogStream",
"logs:PutLogEvents",
]
resources = ["${aws_cloudwatch_log_group.app.arn}:*"]
}
}
data "aws_iam_policy_document" "ecs_assume_role" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
resource "aws_iam_role" "ecs_execution" {
name = "ecs-demo-execution-role"
assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}
resource "aws_iam_role_policy" "ecs_execution" {
name = "ecs-demo-execution-policy"
role = aws_iam_role.ecs_execution.id
policy = data.aws_iam_policy_document.ecs_execution_policy.json
}What the two circuit breaker flags do
The configuration is small:
deployment_circuit_breaker {
enable = true
rollback = true
}Each flag has a separate responsibility.
enable = true allows ECS to detect a deployment that fails to reach a steady state and mark it as failed.
rollback = true tells ECS to recover by returning to the most recent deployment that completed successfully.
Without an eligible previous deployment, automatic rollback has nowhere to return. That is why the first deployment of a service is a different situation from updating an already working service. AWS documentation
How ECS detects a failed deployment
The circuit breaker evaluates failures in two stages: first, ECS checks whether new tasks reach the RUNNING state; once they do, it evaluates container, load balancer, or AWS Cloud Map health checks. A running task alone does not mean the application is healthy.
Repeated task startup failures or replacements caused by failed health checks count toward a deployment failure threshold. When that threshold is reached, ECS marks the deployment as FAILED and triggers rollback if enabled. A single failed health probe does not immediately fail the deployment. AWS documentation
Giving ECS a useful health check
For this Flask application, we can define a container health check in the task definition:
resource "aws_ecs_task_definition" "app" {
family = "ecs-rollback-demo"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
execution_role_arn = aws_iam_role.ecs_execution.arn
cpu = 256
memory = 1024
container_definitions = jsonencode([
{
name = "rollback-demo"
image = var.image_uri
essential = true
portMappings = [
{
containerPort = 5000
protocol = "tcp"
}
]
healthCheck = {
command = [
"CMD-SHELL",
"curl -f http://localhost:5000/health || exit 1"
]
interval = 10
timeout = 5
retries = 3
startPeriod = 20
}
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.app.name
"awslogs-region" = var.aws_region
"awslogs-stream-prefix" = "ecs"
}
}
}
])
}The command runs inside the application container. It calls the local Flask endpoint and returns a nonzero exit code if the request fails. The image must therefore contain curl. A minimal example is:
FROM python:3.12-slim
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir flask
COPY app.py .
CMD ["flask", "--app", "app", "run", "--host=0.0.0.0"]There are two separate failure counters to keep in mind: retries belongs to the container health check, while the circuit breaker evaluates failures at the deployment level. Setting retries = 3 does not mean that the third failed HTTP request immediately rolls back the service.
Deploying a broken release on purpose
Start by building and deploying a working image with a distinct tag, such as:
rollback-demo:v1Wait until its ECS deployment reaches COMPLETED. Confirm that the application responds and that the tasks are healthy.
Next, create a deliberately broken variant. One way to exercise the health-check path is to keep the application running but make /health return an error:
@app.route("/health")
def health():
return "Unhealthy release", 500Build and push this version under another tag:
rollback-demo:v2-brokenUpdate the Terraform image input to point to that image and apply the change.
Distinct tags make the experiment easier to follow: you can identify the working and broken releases without changing the contents behind the same tag.
The new task definition now refers to the broken image. ECS starts its rollout, but the new tasks fail the configured health check. As unhealthy tasks are replaced and the deployment failure threshold is reached, the circuit breaker can trigger.
Observing the rollback
An automatic recovery mechanism is much easier to trust when you can explain what triggered it.
During the experiment, inspect:
- Service events for deployment progress and rollback messages.
- Stopped task details for the reason tasks were replaced.
- CloudWatch logs for application errors.
- Deployment state for the transition from
IN_PROGRESStoFAILED.
ECS also emits deployment state change events through EventBridge, which can be used to notify a team when a deployment fails or a rollback starts. AWS documentation
A useful addition to the demo application is displaying its ECS task definition family and revision. After rollback, that gives you a visible confirmation that requests are being served by the earlier revision.
Conclusion
AWS ECS Rollouts with a Deployment Circuit Breaker provide a practical way to detect failed releases and automatically restore the last successful deployment. Combined with meaningful health checks and CloudWatch logs, this approach helps you recover from deployment failures and understand what went wrong. Testing it with a deliberately broken release lets you verify the recovery process before relying on it in production.
