Skip to content

fix(ops): three apps were monitored blind, and none of them was broken - #424

Merged
github-actions[bot] merged 1 commit into
mainfrom
fix/sweep-follows-host-redirects
Aug 28, 2026
Merged

fix(ops): three apps were monitored blind, and none of them was broken#424
github-actions[bot] merged 1 commit into
mainfrom
fix/sweep-follows-host-redirects

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

The sweep reported aoz-wohnen, revampit and petvity as LIMITED — "no health route, a dead database is invisible". All three have a working health route. The sweep was asking the wrong URL and believing the answer.

That's the one shape a monitor must never get wrong: it looks identical to an app that genuinely has no check, so nobody investigates. Found by running it against production, not by reading it.

Redirects

aoz-wohnen.orangecat.ch 308s to aoz.orangecat.ch; revampit.orangecat.ch to evig.orangecat.ch — same path, canonical host, answering 200. We asked the redirecting name, read 3xx as "no route here", fell back to /, saw the redirect serve, and said LIMITED. Had either database died, the redirect would still have been a 308 and we would still have said "serving".

But a blanket curl -L would be worse than the bug it fixes. petvity 307s /api/health to /login?returnTo=%2Fapi%2Fhealth, so following redirects returns 200 from a login page and reports petvity UP on the strength of a form.

So: follow a redirect only when it keeps the path and merely moves hosts. same_path_redirect decides that; both shapes are pinned by tests.

Wrong path

petvity's /api/health is not a health route at all — it's the pet health-records api, behind auth. Its real check is /api/healthz, answering {"ok":true,"db":true} — the database check we wanted, there the whole time.

HEALTH_PATHS now declares the non-conforming ones (bridge /healthz, petvity /api/healthz) in one table both target lists consult, replacing the third field only the hand-list understood.

annushka

Served on the box with an api on 4030, in no manifest. It won't die with the rest of EXTRA_TARGETS: a static site with no repo can never appear in a manifest whose fields are a repo path and a deploy target — while its api can fall over with the pages still serving. Exactly botsmann's shape.

Result

before after
up 7 12
limited 11 8
down 1 0

The eight remaining were verified to have no health route (/api/healthz and /healthz both 404), not assumed.

Verification

57 unit tests pass; sweep run against production twice. One test pins a bug this PR made on the way: manifest_targets passed the table to awk as a file-argument assignment, which awk applies only when it reaches that argument — long after BEGIN built the lookup. bridge (hand-listed, -v) worked; petvity (manifest) silently kept /api/health. Syntax fine, sweep ran, only the URL was wrong.

Local full verify was blocked by dependency-install contention with another session in this repo; the change is two bash files and CI runs the full chain here.

The sweep reported aoz-wohnen, revampit and petvity as LIMITED — "no health
route, a dead database is invisible". All three have a working health route.
The sweep was asking the wrong URL and believing the answer, which is the one
shape a monitor must never get wrong: it looks identical to an app that has no
check, so nobody investigates.

Two different causes, found by running it against production rather than
reading it.

REDIRECTS. aoz-wohnen.orangecat.ch 308s to aoz.orangecat.ch and revampit to
evig.orangecat.ch — the same path on the canonical host, answering 200. We
asked the redirecting name, read 3xx as "no route here", fell back to `/`, saw
the redirect serve, and called it LIMITED. Had either database died, the
redirect would still have been a 308 and we would still have said "serving".

But a blanket `curl -L` would be worse than the bug. petvity 307s /api/health
to /login?returnTo=%2Fapi%2Fhealth, so following redirects returns 200 from a
LOGIN PAGE and reports petvity UP on the strength of a form. So: follow a
redirect only when it keeps the path and merely moves hosts. same_path_redirect
decides that, and both shapes are pinned by tests.

WRONG PATH. petvity's /api/health is not a health route at all — it is the pet
health-RECORDS api, behind auth. Its real check is /api/healthz, and it answers
{"ok":true,"db":true}: the database check we wanted, there the whole time.
HEALTH_PATHS now declares the non-conforming ones (bridge /healthz, petvity
/api/healthz) in ONE table both target lists consult, replacing the third field
that only the hand-list understood.

Also adds annushka, served on the box with an api on 4030 and in no manifest.
It will not die with the rest of EXTRA_TARGETS: it is a static site with no
repo, so it can never appear in a manifest whose fields are a repo path and a
deploy target — while its api can fall over with the pages still serving.

Live before: up=7 limited=11 down=1. After: up=12 limited=8 down=0.
The eight remaining have genuinely no health route — verified, not assumed.

One test pins a bug this commit made: manifest_targets passed the table to awk
as a file-argument assignment, applied only when awk REACHES that argument,
long after BEGIN built the lookup. bridge worked, petvity silently kept
/api/health. Syntax fine, sweep ran, only the URL was wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018waGt1ieA9TjpscqrbrnGb
@github-actions
github-actions Bot merged commit ffef064 into main Aug 28, 2026
3 checks passed
@github-actions
github-actions Bot deleted the fix/sweep-follows-host-redirects branch August 28, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant