Is Your Cloud Configuration Changing Silently? Understand Infrastructure Drift Before Your Deployment Goes Awry

Teams often realize there is a problem when the next deployment suddenly wants to remove firewall rules, change server sizes, or alter network configurations that are still needed. The cause is not always a bug in the application. It could be that the cloud configuration has changed outside the agreed workflow.

The condition when the actual configuration in the cloud no longer matches the configuration defined in the code or state is called infrastructure drift. This issue commonly occurs when someone changes resources directly from the dashboard, runs manual commands during emergencies, or uses multiple management tools simultaneously.

What is infrastructure drift?

Imagine there is a blueprint for a house, but over several months, the occupants add partitions, move doors, and change electrical installations without updating the blueprint. The house may still stand, but someone reading the old document no longer knows the actual condition.

The same thing happens with cloud infrastructure. Terraform code, CloudFormation templates, or internal documentation describe the expected condition. Meanwhile, the resources in the cloud have the actual condition. If the two differ, drift occurs.

For example:

The database port is opened temporarily for troubleshooting and then forgotten to be closed.
The instance size is manually increased when traffic is high.
Access rules on the storage bucket are changed from the dashboard.
Tags, environment variables, or backup settings are edited by other team members.
Resources are created manually due to urgent needs but are never included in the infrastructure code.

According to HashiCorp documentation, changes outside the Terraform workflow can cause the state to become unsynchronized with the actual infrastructure. In such cases, terraform plan -refresh-only can be used to check for changes without immediately applying changes to the resources. HashiCorp documentation explains this process.

Why is drift dangerous?

Drift is not just a matter of tidiness. It can affect security, costs, and recovery when incidents occur.

1. Deployment may revert important changes

If manual changes have not been recorded in the code, the next deployment may consider those changes undesirable. Infrastructure as Code tools then have the potential to revert resources to the old configuration. From a system perspective, this behavior is correct. However, for the team, the result may appear as if the configuration has “disappeared on its own.”

2. Security gaps may go unnoticed

Open ports, overly permissive buckets, or network rules that weaken access restrictions often arise from small changes. The problem becomes harder to trace if there are no clear change records.

3. Bills can increase without immediate awareness

Manually scaled instances, additional volumes, or temporary resources that are never deleted can add to costs. Without a comparison between the planned condition and the actual condition, such changes can easily slip through.

4. Recovery plans become unreliable

Documentation and code may state that the application has two servers, daily backups, and specific network rules. However, if the production condition has changed, the recovery process from that code may not yield the same system.

How to practically find drift

The first step is not to fix everything immediately. First, find out the differences between the expected condition and the actual condition.

Inventory resources. Document servers, databases, networks, buckets, load balancers, DNS, and other services that are critical to the application.
Determine the source of truth. Choose whether the official configuration is stored in Terraform, CloudFormation, Pulumi, or another system. Do not let the dashboard and repository both be considered primary sources.
Run change checks. For Terraform, use terraform plan in the normal workflow or terraform plan -refresh-only when you want to check the difference between the state and the actual resources.
Check the cloud configuration history. Services like AWS Config can store the history of recorded resource configuration changes, allowing the team to trace when changes occurred and what changed.
Classify each difference. Separate necessary changes, temporary changes, unauthorized changes, and resources that have never been included in the code.

AWS explains that AWS Config can create configuration records for supported resources and store that history. This feature helps answer questions like when resources changed and what configurations were introduced. See AWS Config concepts.

Don’t just “tidy up” everything

A common mistake is to run apply without reading the change plan. If drift has accumulated, a single command can change many resources at once.

Before fixing, make a list of the impacts. Ask:

Are manual changes protecting services from disruption?
Are there important configurations that have not been included in the code?
Does the application need to be placed in maintenance mode?
Will changes disconnect the database, VPN, or third-party services?
Are the state and credentials used correct for the production environment?

If manual changes are indeed necessary, incorporate that decision into the code once conditions stabilize. If changes should not exist, schedule their rollback with a clear rollback plan.

Simple habits to prevent drift

Limit direct changes in production

The dashboard remains useful for observation and emergency conditions. However, permanent changes should go through pull requests, reviews, and approvals. Direct write access to production can be restricted to certain roles only.

Use checks before merging

Every infrastructure change should produce a readable plan. The team does not need to understand all the cloud details, but they should know what resources will be created, changed, or deleted.

Schedule regular checks

For small projects, weekly checks may be sufficient. For systems that change frequently, conduct checks more often through the pipeline. HCP Terraform, for example, provides refresh-only checks to compare the actual configuration with the tracked state. Official explanations are available in HashiCorp documentation.

Document exceptions

Not all temporary changes are mistakes. Importantly, each exception should have a reason, owner, expiration date, and return plan. Simple incident tickets are better than relying on memory.

What does this mean for us?

Infrastructure drift is essentially documentation debt that can turn into operational risk. It is not always visible when the application is running normally, but it usually appears during deployments, security audits, migrations, or disaster recovery.

Start with the most critical service. Compare the code with the actual condition, check the change history, and tidy up the approval workflow. The goal is not to prohibit all manual actions but to ensure that every change can be explained, tracked, and repeated when needed.

The cloud will feel easier to manage when the team not only knows where the resources are but also understands why that configuration was made and which documents can be trusted.

Sources & further reading

– Rio Yotto @rioyotto