Resource recreation with Terraform replace

There are specific situations in Terraform when you need to manually recreate resource even nothing in the code has changed such as incomplete script, resource drift, provisioner failed etc. When resources changed outside of Terraform management of the state file becomes more difficult. This is when resource recreation with Terraform replace comes in handy.

In this blog post you can see few examples of the use cases when terraform replace might help.

Terraform replace use-cases

Example of the command:

terraform apply -replace="aws_instance.web_server"

terraform apply -replace="aws_instance.web_server"

This command is powerful tool that lets you manually recreate a specific resource even in situations when there are no changes in the configuration. It can be very useful if a resource become corrupted or degraded, which means that resource is still there but it is not working properly anymore due some issue. For example an EC2 instance is not responsive due to some operating system failure however Terraform still sees it as healthy. The fix would be to replace resource as in the example above because it forces Terraform to destroy the current, unresponsive EC2 instance and create new one from the exact same Terraform configuration and pretty much setting web server to new clean state.

For sure it occurs that someone modified resource directly (e.g., in AWS Console) and caused mismatch between Terraform state and the real state. Example could be increased web server (EC2 instance) type from t3.micro to t3.medium directly in AWS. This could also be fixed using same command.

Important thing is that some of these scenarios you could simply fix by running regular terraform apply command, however it modifies existing resources in-place and the resource would not be recreated with clean state. On the other hand terraform apply -replace destroys and recreates the resource completely. Additionally provisioners like local-exec or remote-exec in Terraform only run when the resource is created and not when it is updated in place meaning with regular terraform apply it would most likely stop the instance, change the type and then run it again so it would skip the provisioner because the resource is not fully recreated. That is why using flag -replace in such scenarios is useful because it recreates resource and make provisioner run again.

Here is another example:

resource "aws_instance" "nginx_server" {
  ami           = "ami-0c22d132cbfkol1f2"
  instance_type = "t2.micro"

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo apt install -y nginx",
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("~/.ssh/id_rsa")
      host        = self.public_ip
    }
  }
}

resource "aws_instance" "nginx_server" {
  ami           = "ami-0c22d132cbfkol1f2"
  instance_type = "t2.micro"

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo apt install -y nginx",
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("~/.ssh/id_rsa")
      host        = self.public_ip
    }
  }
}

Sometimes a provisioner script fails during the initial terraform apply, leaving the resource only partially configured. Terraform won’t re-run that script automatically unless the entire resource is recreated. For example during first apply the instance is created but for some reason failed to run apt install -y nginx due to some network issue, meaning resource exist but the setup is not complete. This could be fixed by running command:

terraform apply -replace="aws_instance.nginx_server"

terraform apply -replace="aws_instance.nginx_server"

This destroys the current EC2 instance, creates brand new one and re-runs the remote-exec provisioner.

There are scenarios when you want to re-run some script even though nothing failed and the Terraform code hasn’t changed. Here is null_resource example:

resource "null_resource" "deploy_code" {
  provisioner "local-exec" {
    command = "./deploy.sh"
  }
}

resource "null_resource" "deploy_code" {
  provisioner "local-exec" {
    command = "./deploy.sh"
  }
}

This script is used for example to push some files to S3 bucket, maybe sync some code with a server and trigger a build. The issue is when you run terraform apply for the first time deploy.sh will be executed as expected. What could happen is that after some time you update some code or assets outside of Terraform so the configured itself never changed so apply would simply show that everything is up to date. In order to run deploy.sh script you can simply use:

terraform apply -replace="null_resource.deploy_code"

terraform apply -replace="null_resource.deploy_code"

as this will destroy and recreate null_resource. This is reliable way to re-run script even when Terraform configuration didn’t change.

Note that there is another way to achieve this, by running null_resource each time you run terraform apply command by having triggers_replace in place:

resource "null_resource" "deploy_code" {
  triggers_replace = [timestamp()]

  provisioner "local-exec" {
    command = "./deploy.sh"
  }
}

resource "null_resource" "deploy_code" {
  triggers_replace = [timestamp()]

  provisioner "local-exec" {
    command = "./deploy.sh"
  }
}

Depending on you use-case you could also use this approach if it is fine for you that script is automatically re-run on every apply. Also with -replace flag you get minimal output.

Conclusion

Resource recreation with Terraform replace is powerful, precise tool for situations where you need to recreate a resource manually. Whether to re-run provisioners, recover from failure or even undo some changes. This command is very safe to use and it runs through normal Terraform cycle so you will be able to see what exactly will be destroyed and recreated. Also it respects resource dependencies and lifecycle configurations.

If this blog saved you time, support me with a coffee!