A cluster needed a rolling node replacement: swap out old machines for new ones, one at a time, without taking the whole thing down. Nothing unusual about the plan. What turned into a genuine time sink was something that should have been the easiest part: getting a working SSH session to the machines in the first place.
There was a key pair on file, referenced in the deployment notes as the one to use for this cluster. It didn’t work. Permission denied, cleanly and consistently, against every node. There was a second key, sitting on the jump host that operations normally goes through, looking like exactly the kind of thing someone would have set up for this. Also denied. A third candidate, found by process of elimination, worked.
Why “it’s documented” wasn’t the same as “it’s current”
None of this was anyone’s fault in the moment. The documented key had been correct once, presumably when it was written down. Machines get replaced, keys get rotated as part of routine hygiene, and nobody goes back and edits the deployment notes every time that happens, because updating documentation isn’t the part of the task that feels like it matters while you’re doing it. So the notes kept pointing at a key that used to be right, and there was no mechanism forcing anyone to notice it had drifted, because the only time you’d ever find out is exactly this moment: trying to actually connect, under time pressure, during a maintenance window.
The jump host’s own key told a slightly different story. It worked for some machines and not others, which took a second pass to understand: the cluster had grown over time in waves, and different generations of nodes had been provisioned with different key pairs depending on whatever the standard process was at the time each batch was created. There wasn’t one right key for “this cluster.” There were multiple right keys, each correct for a specific subset of nodes, and no single piece of documentation captured that split. Anyone relying on the notes as written would get intermittent, confusing failures that looked like flaky infrastructure rather than what it actually was: several true statements about access, each only partially applicable, written down as one blanket instruction.
That distinction matters more than it sounds. A single wrong key produces a clean, boring failure: nothing works, you go find the right key, done. A set of partially-right keys produces something much harder to diagnose, because early results look like progress. The second key worked on the first two nodes tried, which is exactly enough evidence to conclude “found it” and move on, right up until node three fails with the same permission-denied error the first key gave everywhere. At that point it’s tempting to assume node three is just unhealthy in some other way, because the key already proved itself. Getting past that required treating every node as its own small experiment rather than trusting a pattern from the first couple of successes.
What actually made the rollout work
Once the actual key-to-node mapping was sorted out empirically, by testing systematically rather than trusting any single source, the rest of the replacement went the way it was supposed to: one node out, one node in, verify, repeat. The technical work wasn’t the hard part. Figuring out which of several plausible-looking credentials was actually true, for which subset of machines, was.
The part worth keeping
Access documentation decays quietly and asymmetrically. It doesn’t fail loudly when it goes stale, because nobody’s trying to use it most of the time. It only gets tested under exactly the conditions where you’d most like it to just work: mid-maintenance-window, multiple machines, a clock running. And “multiple valid keys for one system” is a more common state than most documentation accounts for, because it’s easy to write down the answer that was true for the machines you were looking at that day and never revisit it once the fleet quietly diversifies underneath that snapshot. When access notes and actual access disagree, the notes are not automatically the ones telling the truth. They’re just the ones that were true the most recently someone bothered to check.
None of this is really about SSH keys specifically. It’s about any credential or access path that gets written down once, at a moment when it happened to be accurate, and then treated as a permanent fact instead of a snapshot with an expiration date nobody assigned it. The fix isn’t “write better documentation,” because documentation will always lag behind whatever’s actually changing underneath it. It’s building in a habit of verifying access before you need it under pressure, and updating the record the moment you discover it was wrong, rather than fixing the immediate problem and moving on without leaving a trail for the next person who trusts the same notes.