magine you are restoring an Aurora database from an automated snapshot through a Terraform pipeline. The operation takes longer than expected, and the temporary STS credentials expire without being refreshed.
AWS creates the database, but Terraform cannot save the updated state to S3:
Error: Failed to save state
Error saving state: failed to upload state:
operation error S3: PutObject
ExpiredToken: The security token included
in the request is expiredYou open the AWS console and find the database instance there. Expired credentials are one possible cause; network or permission issues can lead to the same outcome: the resource exists, but Terraform’s remote state may not reflect it.
When you check Terraform state:
terraform state show \
'module.database.aws_rds_cluster_instance.this["writer"]'The response is:
No instance found for the given address!How can both be true?
AWS created the resource, but Terraform failed to save that change in its state. If you rerun the pipeline, Terraform may try to create the same resource again and fail because it already exists.
Creating a resource and saving state are separate operations
Terraform uses state to remember which AWS resource belongs to each resource in your code.
For example, state connects these two:
Terraform address:
module.database.aws_rds_cluster_instance.this["writer"]
AWS identifier:
example-test-writerDuring an apply, Terraform makes changes in AWS and saves information about those changes in state. If AWS creates the database writer instance but saving state fails, the database stays there. Terraform of course does not automatically undo changes that already succeeded.
A failed pipeline can therefore leave real changes in AWS, even if they were not saved in state.
Where expired credentials fit in
The pipeline’s temporary AWS credentials may still be valid when Terraform requests database creation. Once AWS accepts the request, it can continue creating the database even if those credentials expire.
Terraform still needs valid credentials to check progress and save the updated state. The AWS provider and the S3 backend can also use different authentication settings: the provider manages infrastructure, while the backend reads and writes state.
An ExpiredToken error during S3:PutObject tells us that Terraform could not authenticate the state upload. It does not tell us whether the database creation succeeded or failed.
Before retrying, restore valid credentials. Rerunning the job only helps if it obtains fresh credentials or can refresh them automatically. Reusing the same expired credentials will cause the operation to fail again.
Confirm that you are checking the correct state
If a resource is missing from state, first make sure you are looking in the right place. You might be in the wrong Terraform directory, using a different backend, or working in another workspace.
Start with:
terraform workspace show
terraform state listCheck that the backend bucket and key, environment, AWS account, and region match the deployment you are investigating.
You can check your AWS CLI identity with:
aws sts get-caller-identityKeep in mind that Terraform may assume a different role through its provider configuration, so the CLI and Terraform might use different identities.
Also check the full resource address. Resources created with for_each include an instance key, such as ["writer"]:
module.database.aws_rds_cluster_instance.this["writer"]Once you confirm these details, compare the state with the resource in AWS. Before making any state changes, make sure no other apply is running against the same state.
Check whether Terraform saved a recovery state file
If Terraform cannot save state to the remote backend, it attempts to save a local copy instead. The error output may point to a file named:
errored.tfstateIn a CI pipeline, this file is stored on the runner and may be lost when the job is cleaned up.
This recovery file may contain several completed changes, so restoring it can avoid importing each missing resource separately. Terraform state recovery
Once backend access is working again, back up the current remote state:
terraform state pull > remote-before-recovery.tfstateCompare it with the recovery file. Make sure the recovery file belongs to the same backend and workspace, and that restoring it would not overwrite changes from a later deployment.
If those checks confirm it is the correct recovery file, upload it:
terraform state push errored.tfstateWhen import is the appropriate solution
Sometimes no usable recovery file remains. For example, the CI runner may already have been removed. If the resource exists in AWS, is missing from the correct state, and should belong to that configuration, importing it can restore the association.
For an Aurora cluster instance:
terraform import \
-var-file="environments/test.tfvars" \
'module.database.aws_rds_cluster_instance.this["writer"]' \
example-test-writerHere:
module.database.aws_rds_cluster_instance.this["writer"]is the resource address in your Terraform configuration.example-test-writeris the existing database instance identifier in AWS.
The matching resource configuration must already exist. Run the command against the correct backend and workspace, using the same environment inputs as the deployment.
Import records the existing instance in Terraform state so Terraform can manage it. The required import identifier depends on the resource type.
Import only handles the resource you specify; it does not recover all changes from the failed apply. If other resources are also missing from state, they need to be checked separately. Each AWS resource should be managed through only one Terraform address and state
Generate a fresh plan after recovery
After recovering state or importing the missing resource, generate a new plan using the intended environment inputs:
terraform plan \
-var-file="environments/test.tfvars" \
-out=tfplanThe recovered resource may require no changes. It may need an update, or the configuration may still request replacement—for example, because you intentionally changed an immutable argument.
Import does not remove that replacement requirement. It gives Terraform the association it needs to evaluate the existing object.
Use the new reviewed plan for the next apply. A plan saved before the failed deployment or state recovery no longer represents the current situation.
Conclusion
A failed Terraform apply does not always mean the AWS changes failed. Before retrying, check what was created and whether Terraform recorded it in state. A recovery state file or an import can restore missing information without deleting the resource. Once state is recovered, generate a fresh plan and review what remains to be done.
