Spinning up IRSA for the AWS Load Balancer Controller on a new cluster, I did the thing that seems like the obviously smart move: instead of writing the IAM policy from scratch, I copied the one already attached to the controller on a separate, existing cluster. That policy had been in live production use for a while, handling real traffic, with nobody reporting a permission problem against it. If any policy on the planet was going to be “complete,” it was that one.
terraform apply went through clean. The IRSA role got created, the ServiceAccount got its annotation, everything reported success.
Then the actual failures showed up, but not right away, and not from where you’d expect.
The gap that only exists between two specific things
Nothing broke immediately after the apply, because the controller pods were still running under the old, broad node role. Annotating the ServiceAccount with a new IRSA role doesn’t retroactively push new credentials into a pod that’s already up; that only happens on the next pod start. So the apply looked clean simply because nothing was actually using the new role yet.
The failures started after a manual kubectl rollout restart of the controller deployment, the step you have to do anyway to get pods to pick up the new IRSA identity, and one that’s easy to treat as a formality once the Terraform side is green. Once the new pods came up actually running under the copied policy, NLB-backed Services started failing to reconcile: AccessDeniedException, FailedDeployModel, the controller unable to finish setting up a load balancer it had no trouble with an hour earlier under the old role.
Why a policy that’s “already proven” wasn’t proof of anything
The copied policy was correct, for the controller version running on the cluster it came from. The new cluster was running a newer version of the AWS Load Balancer Controller, and that version calls elasticloadbalancing:DescribeListenerAttributes while reconciling NLB Services, an ELBv2 API action that didn’t exist, or wasn’t used by the controller, back when the original policy was written. The policy wasn’t missing something because whoever wrote it forgot it. It was missing something because it predates that requirement entirely. “This policy works in production” was a true statement about a different version of the software than the one it was about to be handed to.
A policy JSON file doesn’t carry a version compatibility note. It just sits there, having caused zero problems for as long as you’ve been looking at it, which tells you nothing about whether it covers a newer piece of software calling a newer API. Its correctness is scoped to the exact version it was validated against, and that scope is invisible unless you go check it explicitly.
Two separate blind spots, not one
terraform apply succeeding told me nothing about whether the actual runtime behavior was correct, and that’s worth spelling out, because it’s not just that Terraform can’t see IAM permission errors it didn’t cause. It’s that the moment the gap could even manifest, the rollout restart, happens outside Terraform’s apply step entirely. Terraform updated the annotation; it didn’t restart the pods, and it has no way to know whether or when that restart happens. So there were two separate points where something could go wrong, on two separate operations, and the first one succeeding said nothing about the second one at all. You have to check both, live, after each one, not just after the one you ran through your automation.
What changed about how I do this now
Closing the gap just meant diffing the exact controller version’s documented IAM requirements against the copied policy, finding the one missing action, and adding it. Confirmed by watching AccessDeniedException disappear from the logs once the updated policy propagated.
What changed is the assumption underneath the whole approach. “This policy is already running in production” answers the question “did this policy work for the thing it was written for.” It does not answer “does this policy cover the thing I’m about to attach it to,” and those are only the same question if the software on both ends is running the exact same version, which is worth actually verifying instead of assuming. Separately: for anything involving IRSA, or any change where a Kubernetes-side rollout has to catch up to a Terraform-side annotation, the apply finishing isn’t the finish line. The rollout restart is a second event with its own independent chance to fail, and it deserves its own independent check.