A routine maintenance pass: bump some infrastructure code, run terraform plan expect a small diff and apply it. Instead, the plan came back proposing to change the database engine version on a chunk of production databases, and reading the actual diff, it wasn’t proposing to move them forward. It was proposing to move several of them backward, to a version older than what was actually running.
That’s not a change you want to apply on autopilot just because the plan output is syntactically clean.
Why the plan disagreed with reality
The database version pinned in the infrastructure code hadn’t been touched in a while. The actual running databases had moved on without it. Managed database services routinely apply their own minor-version patches on their own schedule, automatically, for security and stability reasons, entirely independent of whatever your infrastructure-as-code says the version should be. That’s normally invisible and welcome: you get patched without lifting a finger.
It stays invisible right up until someone runs a plan against that same resource for an unrelated reason. Terraform doesn’t know the live version drifted because of an out-of-band automatic patch. It only knows two things: what’s actually running, and what your code says should be running. If those disagree, its job is to propose a change that reconciles them, and it has no way to know which side is the mistake. In this case, the code was stale, not the running database, so “reconcile them” meant “revert a bunch of already-applied security patches,” which is exactly backward from what anyone actually wanted.
Why this is worse than it looks on paper
A version downgrade on a managed database isn’t a metadata change. Depending on the service, it can mean an actual maintenance operation, a restart, or in some cases isn’t even a supported direction at all and would fail partway through, or succeed and quietly strand the database on an unsupported version with no easy path back up. None of that shows up in the plan’s one-line summary. It just says “will be updated,” the same phrasing it would use for a completely harmless change. You have to actually read which direction the version number is moving to catch it, and a plan with dozens of resources changing makes that easy to skim past.
There’s also a timing problem hiding inside that. If a downgrade like this does get applied, the window between “it’s live” and “someone notices something’s wrong” isn’t instant. A database on an older minor version can run for a while without anything obviously breaking, especially if nothing exercises the specific bug or behavior that the newer version had already patched. That means the blast radius of skimming past one line in a plan output isn’t “immediate outage.” It’s “quiet exposure that surfaces on its own schedule, possibly during an incident that has nothing to do with the change that caused it,” which makes it a much harder thing to trace back to its actual cause after the fact.
How this got caught before it went out
The tell wasn’t anything clever. It was reading the plan’s resource-level diff instead of trusting the one-line change count at the top, and noticing that the “before” value on the version field was higher than the “after” value on more than one resource. Once that pattern showed up on the first database, the obvious next step was checking every other resource of the same type in the same plan rather than assuming it was a one-off. It wasn’t. Several of them had drifted the same way, for the same reason, just because they’d all been sitting untouched in the code for roughly the same stretch of time while the platform kept patching them in the background.
What actually happened instead
Every affected file got corrected to match the version the databases had already been quietly patched to, rather than letting the apply pull them backward. Re-running the plan afterward showed zero drift, which is the real confirmation that mattered here, not just “the apply succeeded” but “the code and reality now agree, in the right direction.”
What this generalizes to
Infrastructure-as-code assumes it’s the single source of truth, and most of the time it’s close enough to true that you stop double-checking the assumption. Managed cloud services will happily patch things out from under that assumption without asking, and the only place that shows up is the next plan you run against the drifted resource, phrased identically to any other, harmless change. The habit worth keeping: before applying any plan that touches a version number, direction matters more than the fact that a change exists at all. “It changed” and “it changed for the better” are not the same fact, and a clean plan only confirms the first one.
The same logic applies past version numbers, too. Anywhere a cloud provider is allowed to make automatic changes to a resource that’s also declared in code, that resource can drift in a direction your code doesn’t expect, and the drift stays silent until a plan happens to touch it for some unrelated reason. Treating a clean plan as safe by default, instead of reading every line that actually changes state, is the specific habit that let this one nearly slip through.