feat(ops): watch the certificates, three weeks before they take the box down - #426
Merged
Conversation
…ox down Nothing watched TLS expiry. Caddy renews at 30 days remaining and has never missed, which is exactly why: the check felt unnecessary because it had never been needed. But renewal breaks for reasons no app ever sees — an ACME rate limit, port 80 closed by a firewall change, a DNS record moved — and the first symptom is every site on the box going dark at once. Twenty domains, one failure. The warning exists for weeks; nobody was reading it. Measured today, all twenty are fine: 65-88 days, staggered. That is the moment to add the check, not after the outage that proves it was needed. Reuses the uptime sweep's target list rather than taking a copy — verified identical, 20 targets and 20 certificates. A second list of domains is the gap that hid botsmann for weeks, and it would open here in exactly the same way: a new app registered in apps.conf would be probed for HTTP and silently unprotected against expiry. Thresholds are Caddy's, not arbitrary. ok at 21+ days, because Caddy renews at 30 and has therefore had nine days of attempts; warn from 7 to 20, when renewal should have happened and did not; critical under a week. Unreadable is CRITICAL, never ok: "we could not check the certificate" and "the certificate is fine" must not share an outcome — that conflation is how botsmann's 503 read as healthy for weeks. Daily, not every fifteen minutes. A certificate does not change between sweeps, and twenty TLS handshakes a quarter-hour to relearn the same number is noise that earns nothing — and noise gets muted. No issue/Telegram machinery: fleet-uptime.yml owns that, and a second alerting path to keep in step is not worth it for a check with three weeks of slack. Stated in the workflow rather than left to be discovered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018waGt1ieA9TjpscqrbrnGb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing watched TLS expiry.
Caddy renews at 30 days remaining and has never missed — which is exactly why the check felt unnecessary. But renewal breaks for reasons no app ever sees: an ACME rate limit, port 80 closed by a firewall change, a DNS record moved. The first symptom is every site on the box going dark at once. Twenty domains, one failure. The warning exists for weeks; nobody was reading it.
Measured today, all twenty are fine — 65–88 days, staggered. That's the moment to add the check, not after the outage that proves it was needed.
No second list
It reuses the uptime sweep's targets via
uptime-sweep.sh --certs, and I verified the lists are identical: 20 targets, 20 certificates. A separate list of domains is the gap that hid botsmann for weeks, and it would open here the same way — a new app registered inapps.confwould be probed for HTTP and silently unprotected against expiry.Thresholds are Caddy's, not arbitrary
Unreadable is critical, never ok. "We could not check the certificate" and "the certificate is fine" must not share an outcome — that conflation is how botsmann's 503 read as healthy for weeks.
Daily, not every 15 minutes
A certificate doesn't change between sweeps. Twenty TLS handshakes a quarter-hour to relearn the same number is noise that earns nothing, and noise gets muted.
No issue/Telegram machinery either:
fleet-uptime.ymlowns that, and a second alerting path to keep in step isn't worth it for a check with three weeks of slack. That trade-off is stated in the workflow rather than left to be discovered.Verification
66 unit tests (9 new, pinning every verdict boundary including negative days and unreadable input).
--certsrun against production: 20 domains,ok=20 warn=0 critical=0.