A few months ago I was leading a least-privilege cleanup across several Kubernetes clusters. The goal was straightforward: stop letting every pod inherit one shared node IAM role, and give each workload its own scoped identity through IRSA.
Two environments went as expected. I created the scoped role, annotated the service account, restarted the pods, and confirmed the new identity in the credentials. Done.
The third environment did not.
The Assumption
Before touching production I ran a 90-day CloudTrail audit. I wanted to confirm which roles were actually being assumed, so I would not accidentally scope down something still in active use.
I expected to see a healthy number of AssumeRoleWithWebIdentity calls from the load balancer controller using its IRSA role. What I got was zero. Not low usage. Zero calls in 90 days.
I assumed I had made a mistake in the query and ran it again. Same result.
Pulling the Thread
I checked the ServiceAccount. There was no IRSA annotation. It had never been added.
The controller had been running the entire time on the node’s IAM role, the same broad role the whole cleanup project existed to eliminate. The IRSA role itself existed. It had a policy attached and looked correct in the IAM console. But the actual wiring on the Kubernetes side had never happened.
Going by the role’s creation date, this situation had lasted for the better part of a year.
The Fix Was Almost Nothing
The actual change was two lines:
YAML
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/<irsa-role-name>The annotation went through the normal process: Helm values in Terraform, PR review, pipeline apply. No manual changes against the cluster.
The only extra step was a rollout restart. Kubernetes does not push a new IRSA identity into already-running pods. The annotation is only picked up when a pod starts. After the restart I confirmed the new pods were presenting the IRSA role instead of the node role.
I also verified the scoped policy contained everything the controller needed before applying the change. No point trading excessive access for missing permissions.
The investigation took far longer than the fix.
Why This Is Easy to Miss
If you only look from the AWS side, a scoped role that exists, has a reasonable policy, and has a matching name reads as “handled.” You have to check the actual ServiceAccount manifest on the Kubernetes side to see that it was never connected. Nothing throws an error. The pod simply continues working with more permission than it should.
A year after the original “migration” ticket was closed, almost nobody goes back to verify.
What Changed for Me
I no longer treat “the IAM role exists and looks correct” as proof that the workload is using it. The check is now two-sided:
- Does the scoped role exist in AWS?
- Does CloudTrail actually show it being assumed?
If I cannot confirm the second part, the work is not done.
CloudTrail verification also stopped being optional due diligence. It became a required step before any least-privilege work. This was not a case of someone leaving a door open. It was a case of everyone, including me, being convinced the door was already locked.
Conclusion
Configuration intent and runtime reality can diverge quietly. In this case everything looked correct in IAM, yet the workload had never used the scoped role.
The cheapest way to catch this class of problem is to verify both sides: the identity exists, and it is actually being assumed.