We wanted FluentBit picking up systemd journal logs: kubelet, containerd, the usual host-level stuff that doesn’t come from a container’s stdout. The Helm chart supports it, with an additionalInputs block and a systemd input plugin mounted against the host’s journal directory. I wrote the config, deployed it, and checked the rendered ConfigMap. The systemd INPUT section was there, formatted correctly, volumes mounted where they needed to be.
FluentBit came up healthy. No systemd logs showed up anywhere downstream. I checked the pod’s own startup logs for the line FluentBit normally prints when a plugin initializes, and found nothing. No [input:systemd:systemd.0], no error about it either. As far as the running process was concerned, that input plugin didn’t exist.
Ruled out the obvious things first, because they’re usually right
I re-checked the YAML structure against the chart’s docs. Correct. I re-checked that the ConfigMap actually mounted into the pod, verified by exec’ing in and reading the file directly off disk. It did. I bumped the chart to the next available patch version, on the theory that maybe this specific release had a config-rendering bug. Same result: correct config, silent non-load.
At that point the question stopped being “is my config right,” because it clearly was, and became “does this actual running binary have this plugin compiled into it at all.”
Chart version and binary version are two separate numbers
They looked like the same thing from the outside, and they weren’t. The Helm chart’s version number tracks the chart itself: the templates, the values schema, the packaging. It doesn’t move in lockstep with the FluentBit binary version baked into the container image the chart deploys. The entire 0.1.x line of this particular chart, through several patch releases, shipped the exact same underlying FluentBit binary: version 1.9.10. FluentBit’s systemd input plugin, the actual C code that reads from the journal, wasn’t compiled into that build at all. It’s a feature gated behind build flags, and 1.9.10 simply wasn’t built with it on.
So every patch bump within that chart line was cosmetic as far as this problem was concerned. The config was accepted, rendered, and mounted correctly every time, straight into a binary that had never once been capable of acting on it. There was no error, because from FluentBit’s point of view there wasn’t anything to error about. It read a config section for a plugin type it doesn’t have, and the failure mode for that, in this version, is silently skipping it rather than refusing to start.
Jumping the chart line solved it
Moving to the chart’s next major line, 0.2.0, brought in FluentBit 4.2.2, a binary built with the systemd plugin compiled in. Same config block, unchanged. The moment the underlying binary actually had the capability, the exact same YAML that had been silently doing nothing for several versions started working, and [input:systemd:systemd.0] showed up in the startup log on the very first pod restart.
The part worth generalizing
The instinct when a “supported” feature doesn’t work is to suspect your own config first, and that’s the right instinct maybe 90% of the time. Most silent failures really are a formatting mistake or a missing mount. But a chart, an operator, or any other packaging layer advertises features at the level of configuration schema, not at the level of what the underlying binary can actually do. The chart happily accepts a systemd input block because its schema allows it. Schema validation and runtime capability are two entirely different layers, and a packaging tool checking the first tells you nothing about the second.
The concrete habit that comes out of this: when a documented feature silently no-ops, check the actual binary version inside the running container before spending more time re-reading your own YAML. Not the chart version, not the appVersion label, the literal version string the process reports. chart 0.1.32 == FluentBit 1.9.10 and chart 0.2.0 == FluentBit 4.2.2 was the one fact that made the whole thing click, and no config file was ever going to tell me that.