Why Your Staging Environment Drifted (and How to Stop It)

Your staging environment has not matched production in eighteen months. Nobody can quite say when it stopped. The person who set it up left. Someone ran a database migration in the console. A library upgrade happened on prod but not on staging. Now your tests pass in staging and fail in production, so you have stopped trusting staging.

This is not unusual. It is also not inevitable.

Why it happens#

Staging drifts because infrastructure is treated as state instead of code. You click things in a cloud console. You ssh into a box and run commands. You upgrade the database here but forget about there. There is no diff, no version control, no pull request. If you need to reproduce it, you either remember what you did (you will not) or you rebuild it from scratch (which fails halfway through).

DevOps is the practice of treating infrastructure like code—versioned, reviewed, reproducible. But most teams do not do it. Instead they have environments that diverge quietly until they are so different they might as well be separate systems.

The result: staging is not a test of production deployment. It is a random system that happens to exist.

The moment you realize#

A deploy that works perfectly in staging explodes in production. The error is something like "can't find the library" or "wrong database driver version." You investigate and find the environments differ in a way that nobody documented.

Then someone says, "Well, that is why we test in production."

And then something breaks in production, and everyone learns at the same time.

The fix#

Infrastructure as Code (IaC). All your infrastructure—networks, databases, load balancers, security groups, environment variables, the lot—lives in version control as declarative definitions (usually Terraform or CloudFormation).

When you want to change something, you:

1. Edit the code
2. Review the diff in a pull request
3. Run terraform plan to see what will actually change
4. Approve the PR
5. Run terraform apply to change it
6. It is recorded in git history

Now your staging and production environments are not "set up similarly." They are built from nearly identical code that differs only in parameters (instance count, database size, etc.).

When someone asks "why is staging different from production," the answer is in git blame.

When you want to rebuild staging, you run one command and it rebuilds from the definition. Identical.

What changes#

Before: Staging and production are hand-built systems that diverged years ago.

• Deploying to staging is a partial test because the environments do not match
• When production breaks, staging does not help because it is different
• On-call person at 2am is guessing at what might be wrong

After: Staging and production are built from the same infrastructure code.

• Deploying to staging is a real test because the environments match (except for scale)
• When production breaks, staging can reproduce it because they are built the same way
• On-call person can look at the Terraform plan that was merged last week and start there

Where to start#

If you have existing infrastructure running, you do not need to blow it up and rebuild. You import the resources into Terraform incrementally—a database one week, the load balancer the next. Each import is a pull request. By the time you are done, every resource is under version control and you can stop hand-configuring things.

That is what we do. It is not complicated. It is boring, which is exactly the point.

The boring part is what stops environments from drifting.