We were tightening up IAM on a Kubernetes cluster, moving log-shipping agents off the big shared node role and onto their own scoped IRSA roles, one component at a time. Standard least-privilege cleanup. For the CloudWatch log shipper, that meant writing a policy that only allows logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents against the exact log group ARN pattern the agent was actually configured to write to. Nothing exotic: read the Helm values, see the logGroupName template, derive the ARN, lock the policy down to that.
Applied it. Log ingestion kept working. Looked done.
A few days later, a burst of AccessDeniedException on logs:CreateLogGroup showed up in CloudTrail, for a log group name that had never appeared anywhere in the Helm values, the ConfigMap, or any doc describing how this agent was set up.
Reading the config wasn’t enough
The log group name the agent was configured to use follows a template that includes the pod’s namespace, something like /aws/eks/<cluster>/<namespace>/.... That’s the resource the scoped policy covered, correctly, because that’s what the configuration says will happen.
What the configuration doesn’t say is what happens when that template can’t resolve, when a given log record doesn’t carry a usable namespace tag for whatever reason (a record from a pod that’s already gone, something during a rollout, edge cases in how the tagging filter runs). aws-for-fluent-bit doesn’t drop that record or error out loudly. It falls back to a hardcoded default log group name, baked into the binary, and writes there instead. That fallback name doesn’t show up anywhere: not in the Helm chart’s values schema, not in the rendered ConfigMap, and there’s no flag that says “by the way, sometimes I ignore your logGroupName setting and use my own.” You only find out it exists by going and reading the agent’s own source or its GitHub issues, not by reading your own configuration.
So the policy was airtight for the log group the config describes, and had a zero-tolerance gap for the log group the binary sometimes actually decides to use, a resource nothing in the configuration gave any reason to expect.
Why it took a few days to show up
This isn’t the kind of thing that fails on the first request. It only fires when the fallback path actually gets hit, which depends on some specific-enough condition in how records get tagged as they flow through the pipeline. Most traffic went through the happy path and landed exactly where the policy expected. The failing traffic was a small enough fraction that it took a few days of accumulated denied-write attempts before it surfaced as a pattern instead of a one-off blip, which is the kind of thing you’d normally notice by looking at your logs, except these were exactly the log lines failing to get written.
The fix and the actual lesson
Adding the fallback log group’s ARN (and its :* wildcard for the log streams under it) to the scoped policy, right next to the properly-configured one, took care of it. Confirmed by watching the CloudWatch IncomingLogEvents metric resume immediately after the policy update went live.
The part worth remembering isn’t the fix, it’s what it means for how you write a least-privilege policy in the first place. “Read the config and scope the policy to exactly what the config says” feels like the responsible, careful way to do this. It’s also not sufficient, because software has opinions of its own that don’t show up in configuration: default behaviors, fallback paths, edge-case handling that exists precisely for the situations your configuration doesn’t cover. If you only scope permissions to what you can see in the values file, you’ve scoped them to the happy path, and the whole point of least privilege is making sure the unhappy path doesn’t just fail open onto a shared role with way more access than it needs.
Before locking down permissions for any log shipper, metrics agent, or anything else with a “here’s where I write” setting, it’s worth a few extra minutes checking whether that thing has a fallback destination hardcoded somewhere in its source or its known-issues list. It’s a small check, and it’s the difference between a policy that’s actually complete and one that’s complete only for as long as nothing unusual happens.