We had a recurring nginx problem: a reverse proxy in front of several AWS Application Load Balancers would occasionally stop passing traffic through cleanly, throwing upstream timed out (110: Connection timed out) while connecting to upstream errors for a domain that had been working fine minutes earlier. A plain systemctl reload nginx always fixed it instantly. That combination — instant fix, but only if you know to run it — is exactly what you’d automate away with a watchdog. So we did. And the first version of that watchdog had a gap that let a multi-hour outage happen right past it.
The underlying bug
Same root cause behind all of these incidents: nginx was configured with a static upstream block pointing at an ALB by hostname. Nginx resolves that hostname to an IP once, when the worker starts, and then holds onto it — no periodic re-check, no TTL awareness. AWS rotates the IPs behind an ALB’s DNS name as part of normal scaling and maintenance. So every so often, the IP nginx cached stops being valid, and every request nginx tries to send there hangs until it times out.
The fix for one specific timeout is trivial: nginx -t && systemctl reload nginx forces fresh DNS resolution. The reason it’s not automatic by default is the same reason a blunter fix — a resolver directive with a variable-based upstream that re-resolves on every request — was tried and rejected earlier: the backend held user sessions in memory per task, with no shared session store. Re-resolving mid-session and routing a user’s next request to a different backend task meant silently invalidating their session. So “always re-resolve” was off the table, and “reload only when something’s actually stale” needed some kind of trigger.
Version 1: reload on error spikes
The first watchdog was straightforward: a script polling one specific error log every two minutes, counting matches for the upstream-timeout pattern, and reloading if it saw five or more within a 120-second window:
# simplified logic of v1
ERRORS=$(grep -c 'upstream timed out (110) while connecting to upstream' \
/var/log/nginx/example-error.log)
if [ "$ERRORS" -ge 5 ]; then
nginx -t && systemctl reload nginx
fi
Plus a 10-minute cooldown between reloads, so a burst of errors couldn’t trigger reload after reload after reload — nginx reloading itself in a loop is its own kind of self-inflicted instability. It logged to syslog. It worked, in the sense that it caught the exact incident it was built for and nobody had to get paged at 2am to run one command.
It also only watched one log file, for one domain, because that’s the incident that prompted building it.
The gap
Months later, a completely different domain behind the same nginx box went down for a couple of hours. Same root cause — ALB rotated an IP, nginx cached the stale one — but this domain’s traffic landed in a different log file, one the watchdog had never been told to look at. The script was running the whole time, ticking every two minutes, finding zero matching errors in the one file it knew about, and reporting nothing wrong. A large batch of upstream timeout errors accumulated in the log file it wasn’t watching before anyone noticed the site was actually down.
It also wasn’t a clean, obvious “site is down” signal, which is part of why it took a while to surface. An ALB’s DNS name resolves to a pool of IPs — one per Availability Zone, typically — and AWS doesn’t rotate all of them at once. So when one IP in that pool goes stale, nginx’s upstream still round-robins across the whole pool: some requests land on the still-good IP and load fine, some land on the stale one and hang. From the outside that looks exactly like “the page is broken, then it isn’t, then it is again if you refresh” — not a hard outage, closer to a coin flip on every load. That’s a worse thing to triage than a clean failure, because a page that “mostly works” doesn’t reliably trip anything threshold-based, and users tend to just retry and move on instead of reporting it — right up until enough of them don’t, and it registers as an actual incident.
That’s a worse failure mode than not having a watchdog at all. No monitoring means you know you’re flying blind. A watchdog that’s quietly only covering a fraction of what it implies it covers gives you false confidence — you stop manually checking the thing you built automation specifically so you wouldn’t have to check.
Version 2: cover everything, lower the bar, drop the cooldown
The rewrite widened scope to every domain the box was proxying — over twenty of them, each with its own error log — instead of the one that happened to have caused a problem before. That’s the actual fix for the gap; everything else was tuning:
- Threshold dropped from 5 errors/120s to 2 errors/60s. The gap incident had produced a steady trickle of errors rather than a sharp spike — because only one IP in the pool had gone stale, only a fraction of requests were failing at any given moment. A threshold tuned for “everything is broken” doesn’t reliably catch “roughly half of requests are broken,” which is the more common real-world shape this problem takes.
- Cooldown removed entirely. The 10-minute cooldown made sense for a one-off DNS blip that a single reload fixes. It’s actively harmful for a persistent issue that needs repeated reload attempts every cycle until it clears.
- Notification moved from syslog-only to an actual email alert, because “the fix worked so nobody noticed” and “nobody is watching syslog closely enough to catch a coverage gap” are the same failure mode from two different angles. An alert you have to go looking for isn’t meaningfully different from no alert.
The part that almost undermined the fix anyway
The new script got written, tested in dry-run, and “deployed.” Except the deployment step created a backup of the old script and stopped there — the new one never actually got copied into place. The old, single-domain, no-email version kept running, undetected, for hours, because the deploy process reported success without anyone re-checking what was actually live on disk afterward.
Caught by going back and verifying line count and file size against what should have been deployed, and finding it matched the backup instead of the intended new script. Redeployed, and this time confirmed by actually checking the live file’s size and content — not just checking that the deploy command exited zero.
This is worth sitting with for a second: a script built specifically to close a monitoring gap almost stayed unmonitored itself, for the same underlying reason as the original bug — something reported success without anyone verifying the actual end state. “The deploy ran” and “the new behavior is live” are not the same claim, and treating them as interchangeable is exactly how the first gap happened too.
Version 3: telling you which domain, not just that something’s wrong
Once the wider watchdog was live and catching real events, the alerts it sent were technically correct and practically annoying: “3 errors in this log file” tells you almost nothing at 3am. You still have to go open the log yourself to find out which of twenty-plus domains is actually affected before you can decide if it matters.
Fix was small — extract the domain from the nginx error log’s own server: field and put it in the alert:
# before: "/var/log/nginx/example-error.log: 3 errors"
# after: "/var/log/nginx/example-error.log: 3 errors (domains: docs.example.com)"
DOMAIN=$(awk -F'server: ' '{print $2}' "$LOGFILE" | cut -d',' -f1 | sort -u)
Not a functional change to when it reloads — purely about cutting the seconds between “got an alert” and “know whether to care,” which matters more than it sounds like when the alert usually turns out to be the system successfully healing itself and you just want to confirm that quickly and move on.
What this actually teaches
Four separate lessons stacked on top of each other here, and each one is a different flavor of the same root problem — trusting that something is working because nothing told you otherwise:
- Fixing the instance you’re looking at instead of the class of problem it belongs to just relocates the next incident. The first watchdog was built to solve “this one domain has this one failure mode” — a point fix for a point complaint. But the actual bug wasn’t specific to that domain at all; it was a property of the whole nginx box, true for every domain it proxied to an ALB by hostname. A point fix on top of a systemic cause buys you one quiet postmortem and nothing else — the same failure is still live everywhere you didn’t happen to look. The version that actually held up was the one written to the scope of the real cause (every domain on that box sharing that upstream pattern), not the scope of the ticket that prompted it.
- Monitoring scoped to “the thing that already broke” doesn’t cover “the next thing that will break the same way.” This is the same trap from a different angle: even once you’re building the right kind of coverage, it’s easy to still only wire up the specific case in front of you. If ten things share a root cause, monitoring one of them buys you confidence about the other nine that the evidence doesn’t support.
- A deploy that reports success and a deploy that actually happened are two different facts. Verify the second one directly — check the artifact that’s actually running, not the exit code of the thing that was supposed to put it there.
- An alert nobody has to act on within the same minute they read it is worth almost as little as no alert. If the fix is genuinely automatic and the alert is just for awareness, make the alert answer the first question you’d ask, not just confirm that a question exists.