We build custom machine images: install the OS updates, bake in the endpoint security agent, snapshot it, and use that image to launch new servers going forward. Standard practice, saves a install/configure/register cycle on every new box. After one of these image refreshes, a few newly launched servers started showing up strangely in the security console: their activity was being attributed to a completely different, already-existing machine. Not a naming collision, not a display bug. The console genuinely believed one physical identity was moving between multiple running servers.
Following where the identity actually lives
The security agent doesn’t identify a host by hostname or IP. It generates a unique agent ID the first time it starts up, and that ID is what the security console actually tracks. Hostnames change, get reused, get typo’d. The agent ID is supposed to be the one thing that’s stable and unique per install.
Baking the agent into a golden image works fine for the install part. What it doesn’t automatically handle is that unique ID: it gets generated once, when the agent first starts on the machine you’re building the image from, and then it just sits there in the agent’s local state, waiting to be picked up. Snapshot that machine into an image without clearing it first, and every new server launched from that image boots up with the agent already “registered,” carrying the exact same ID as the box the image was built from. From the console’s point of view, that’s not five new machines. It’s the same one machine, mysteriously running in five places at once, and whichever one checks in most recently is the one the dashboard shows as “current.”
Why it took actual incident response to notice
None of this throws an error anywhere in the normal build pipeline. The agent installs cleanly. It runs. It reports in. Every health check you’d normally run against a freshly baked image says everything is fine, because from a pure “is the agent running” perspective, it genuinely is. The problem only becomes visible from the other side, inside the security console, and only if you’re specifically looking at identity rather than status. A dashboard showing “agent online, last check-in 2 minutes ago” looks identical whether that’s one legitimate ID or five servers sharing one.
It surfaced because someone was investigating an actual alert and the trail led to a server that, according to the console, had been decommissioned weeks earlier. It hadn’t. A newer server, built from the tainted image, had inherited its identity and was quietly answering to its name.
The fix, and the part that’s easy to skip
The agent has a documented command to clear its local identity before a snapshot, forcing it to generate a fresh one the next time it boots. It’s a single step in a build process that already has a dozen other steps, and it’s exactly the kind of thing that’s invisible in a code review of the build script: nothing about skipping it looks wrong, because the image still builds, still boots, still runs the agent successfully. The absence of one command doesn’t show up as a missing line in a diff you’d think to look for. It shows up as a coincidence you’d have to already be suspicious of something to catch.
The broader habit that came out of this: anything you bake into a golden image that generates its own persistent unique identifier on first run needs that identifier explicitly reset as its own verified step before the snapshot, not just assumed to be handled because “the install script ran.” A successful install and a clean identity are two different guarantees, and only one of them gets exercised by the fact that the service starts up and looks healthy.