fix(ops): three apps were monitored blind, and none of them was broken - #424
Merged
Merged
Conversation
The sweep reported aoz-wohnen, revampit and petvity as LIMITED — "no health
route, a dead database is invisible". All three have a working health route.
The sweep was asking the wrong URL and believing the answer, which is the one
shape a monitor must never get wrong: it looks identical to an app that has no
check, so nobody investigates.
Two different causes, found by running it against production rather than
reading it.
REDIRECTS. aoz-wohnen.orangecat.ch 308s to aoz.orangecat.ch and revampit to
evig.orangecat.ch — the same path on the canonical host, answering 200. We
asked the redirecting name, read 3xx as "no route here", fell back to `/`, saw
the redirect serve, and called it LIMITED. Had either database died, the
redirect would still have been a 308 and we would still have said "serving".
But a blanket `curl -L` would be worse than the bug. petvity 307s /api/health
to /login?returnTo=%2Fapi%2Fhealth, so following redirects returns 200 from a
LOGIN PAGE and reports petvity UP on the strength of a form. So: follow a
redirect only when it keeps the path and merely moves hosts. same_path_redirect
decides that, and both shapes are pinned by tests.
WRONG PATH. petvity's /api/health is not a health route at all — it is the pet
health-RECORDS api, behind auth. Its real check is /api/healthz, and it answers
{"ok":true,"db":true}: the database check we wanted, there the whole time.
HEALTH_PATHS now declares the non-conforming ones (bridge /healthz, petvity
/api/healthz) in ONE table both target lists consult, replacing the third field
that only the hand-list understood.
Also adds annushka, served on the box with an api on 4030 and in no manifest.
It will not die with the rest of EXTRA_TARGETS: it is a static site with no
repo, so it can never appear in a manifest whose fields are a repo path and a
deploy target — while its api can fall over with the pages still serving.
Live before: up=7 limited=11 down=1. After: up=12 limited=8 down=0.
The eight remaining have genuinely no health route — verified, not assumed.
One test pins a bug this commit made: manifest_targets passed the table to awk
as a file-argument assignment, applied only when awk REACHES that argument,
long after BEGIN built the lookup. bridge worked, petvity silently kept
/api/health. Syntax fine, sweep ran, only the URL was wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018waGt1ieA9TjpscqrbrnGb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sweep reported aoz-wohnen, revampit and petvity as LIMITED — "no health route, a dead database is invisible". All three have a working health route. The sweep was asking the wrong URL and believing the answer.
That's the one shape a monitor must never get wrong: it looks identical to an app that genuinely has no check, so nobody investigates. Found by running it against production, not by reading it.
Redirects
aoz-wohnen.orangecat.ch308s toaoz.orangecat.ch;revampit.orangecat.chtoevig.orangecat.ch— same path, canonical host, answering 200. We asked the redirecting name, read 3xx as "no route here", fell back to/, saw the redirect serve, and said LIMITED. Had either database died, the redirect would still have been a 308 and we would still have said "serving".But a blanket
curl -Lwould be worse than the bug it fixes. petvity 307s/api/healthto/login?returnTo=%2Fapi%2Fhealth, so following redirects returns 200 from a login page and reports petvity UP on the strength of a form.So: follow a redirect only when it keeps the path and merely moves hosts.
same_path_redirectdecides that; both shapes are pinned by tests.Wrong path
petvity's
/api/healthis not a health route at all — it's the pet health-records api, behind auth. Its real check is/api/healthz, answering{"ok":true,"db":true}— the database check we wanted, there the whole time.HEALTH_PATHSnow declares the non-conforming ones (bridge/healthz, petvity/api/healthz) in one table both target lists consult, replacing the third field only the hand-list understood.annushka
Served on the box with an api on 4030, in no manifest. It won't die with the rest of
EXTRA_TARGETS: a static site with no repo can never appear in a manifest whose fields are a repo path and a deploy target — while its api can fall over with the pages still serving. Exactly botsmann's shape.Result
The eight remaining were verified to have no health route (
/api/healthzand/healthzboth 404), not assumed.Verification
57 unit tests pass; sweep run against production twice. One test pins a bug this PR made on the way:
manifest_targetspassed the table to awk as a file-argument assignment, which awk applies only when it reaches that argument — long afterBEGINbuilt the lookup. bridge (hand-listed,-v) worked; petvity (manifest) silently kept/api/health. Syntax fine, sweep ran, only the URL was wrong.Local full
verifywas blocked by dependency-install contention with another session in this repo; the change is two bash files and CI runs the full chain here.