An AWS deprecation notice landed in the inbox: Systems Manager (SSM) was retiring the ec2messages endpoint that Run Command and Session Manager had used for years, in favor of ssmmessages. Instances running an SSM Agent below 3.3.3598.0 wouldn’t be able to speak the new protocol and would lose remote command / remote shell access after the cutover date.
That kind of notice always starts as a version-number problem. This one turned into a guided tour of exactly how much of the fleet’s access model nobody currently owned had opinions about, made by people who weren’t around anymore, that everything had quietly kept working on top of for years.
Step one looked simple
SSM needs two things to work on a box: an agent recent enough to speak the current endpoint, and an IAM role with the permissions to match. Checking exposure meant going through the whole fleet – well over a hundred instances spanning several different infrastructure types, EC2-based clusters, standalone Linux boxes, one Windows server – and asking, per instance: is the agent new enough, and does the attached role actually carry AmazonSSMManagedInstanceCore (the current policy, which covers ssmmessages), or is it still sitting on AmazonEC2RoleforSSM (a policy AWS itself deprecated back in 2017, scoped only to the old endpoint).
Most of the fleet was fine or one small agent bump away from it. But a couple of IAM roles – attached to a stateful clustered service, the kind of infrastructure that just runs and nobody touches – were still on the 2017 policy. They weren’t broken. describe-instance-information showed them Online. Run Command worked. Session Manager opened a shell without complaint. Whoever attached that policy did so correctly, years before ssmmessages existed, and then the role just… kept existing, exactly as configured, through every reorg and handoff since, because nothing about it ever threw an error.
The actual command to fix it is one line:
aws iam attach-role-policy \
--role-name <role-name> \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Run that against the couple of affected roles, roll out an agent update in batches: staging, then production, then anything running a stateful cluster held for a maintenance window – and step one is done. This part was genuinely quick.
Step two: do it “properly,” and hit the second landmine
Manually attaching an IAM policy with the CLI works, but it’s a one-off, invisible to code review, and the kind of thing that gets forgotten the next time someone rebuilds that role from Terraform and undoes it without knowing why it was there. So the obvious next move was: put the fix into the actual Terraform module that manages these roles, and let it roll out properly.
That’s where it stopped being quick. The module hadn’t been bumped to a newer version in a long time, and a newer version was exactly what was needed to carry the IAM fix. Running terraform plan against the bump showed something nobody wanted to see attached to a stateful production cluster: a destroy-and-recreate of the storage volumes backing those nodes. Not a modification – a replacement. The module had, at some point between the old version and the new one, changed how it addressed its resources internally, from a numeric list index to a named key. Terraform doesn’t know those two addressing schemes refer to the same underlying resource unless you tell it – which means, from Terraform’s point of view, the volume at the old address doesn’t exist anymore, and the volume at the new address needs to be created from scratch. Applying that plan verbatim would have deleted and rebuilt the data volumes of a live cluster to fix an IAM policy.
That PR got closed, not merged. Doing the migration safely needs terraform state mv to manually remap every affected resource address from the old index-based scheme to the new key-based one, before the version bump ever gets applied – its own project, unrelated to SSM, that nobody had prioritized because the old module version wasn’t causing visible problems either. Same shape of legacy as the IAM policy: something set up correctly years ago, quietly frozen in place ever since, that only becomes a problem the moment something forces you to touch it.
So the actual fix that shipped was the manual CLI command from step one, deliberately left as a manual change outside Terraform. Not because it’s the better long-term answer – it isn’t, it’s a policy attachment that exists nowhere in version control and depends on someone remembering it’s there – but because the “correct” path was blocked by a second piece of inherited debt that was its own multi-step project, and an IAM policy sitting outside Terraform is a much smaller risk than an accidental storage-volume rebuild on a live cluster.
Why none of this shows up until you go looking
Both of these – the 2017 IAM policy and the frozen Terraform module – are invisible from the outside for the same reason. A working system tells you it’s working. It doesn’t tell you whether it’s working because it’s current, or working because nobody’s changed anything around it recently enough for the gap to matter yet. AmazonEC2RoleforSSM and AmazonSSMManagedInstanceCore produce identical green checkmarks in the console for as long as AWS keeps supporting both endpoints. An old Terraform module plans clean and applies clean, right up until a version bump exposes an addressing change nobody flagged at the time it happened. Neither one throws a warning on its own. Both waited for something external – the same deprecation notice, twice over – to force the question.
What actually generalizes here
Inheriting infrastructure you didn’t build means inheriting every decision that was correct the day it was made and never revisited since, and you generally don’t find out which ones those are until an external deadline makes you check. When that happens, the “proper” fix and the fast fix aren’t always the same fix, and picking the fast one deliberately – with a clear note about why, and what debt it’s leaving behind – is sometimes the more honest choice than forcing the proper one through and hoping the collateral damage doesn’t matter. The failure mode to avoid isn’t leaving debt in place. It’s leaving it in place silently, the same way the previous owner did.