fix(tools): tell an upstream outage from a failed check

[VizzleTF]

Problem

Run 34836792047, check / build on a pull request, 2026-09-14, during a GitHub incident:

curl: (22) The requested URL returned error: 500     (six times, from tools/fetch.sh)
##[error]Process completed with exit code 22

fetch.sh already retried: --retry 5 --retry-delay 2 --retry-all-errors, about ten seconds. Then it exited 22, which is indistinguishable from a checksum or signature failure, and a manual rerun passed. owfeed's CLI contract separates the two (exit 7: a check failed, never retry; exit 8: upstream outage, safe to retry). The shell here did not.

Measurements

curl, against a local stub that answers a chosen status. Same result on curl 8.7.1 (macOS) and curl 8.5.0 (ubuntu:24.04, the runner's version):

case	exit	%{http_code}	stderr
404 / 403	22	404 / 403	curl: (22) The requested URL returned error: 404
408 / 429 / 500 / 502 / 503 / 504	22	the status	same shape
connection refused	7	000	Failed to connect ... Couldn't connect to server
unresolvable host	6	000	Could not resolve host
timeout	28	000	Connection timed out
--retry 2 on 404	22	404	asked once
--retry 2 on 429 / 500 / 503, DNS, refused (--retry-connrefused)			asked 3 times
--retry 2 --retry-all-errors on 404	22	404	asked 3 times

So under -f the exit code cannot tell a 500 from a 404; only -w '%{http_code}' can. --retry-all-errors, which fetch.sh used, re-asks for a 404.

gh 2.99.0 exits 1 for everything, so only the text tells an outage from an answer:

case	stderr
release view, 5xx / 429 (stub via github.localhost)	HTTP 503: <message> (http://api.github.localhost/repos/o/r/releases/latest)
gh api, 5xx	gh: <message> (HTTP 503)
unreachable proxy	Get "https://api.github.com/...": proxyconnect tcp: dial tcp 127.0.0.1:1: connect: connection refused
unresolvable host	error connecting to nonexistent.invalid check your internet connection or https://githubstatus.com
missing tag / missing repo (real API)	release not found
pattern matches nothing (real API)	no assets match the file pattern
gh api missing repo (real API)	gh: Not Found (HTTP 404)

gh made exactly one request per call in every case; it does not retry.

Design

tools/net.sh, one helper, run as a command (nothing under tools/ sources another script):

• tools/net.sh get <url> <dest> runs curl without --retry and loops itself: 5 attempts, delays 5/10/20/40 s (about 75 s).
  • 408, 425, 429, 5xx, and curl exits 5/6/7/16/18/28/35/52/55/56/92 → retried, then exit 8 with !! upstream outage: … and, in Actions, an ::error title=Upstream outage (exit 8, safe to rerun):: annotation.
  • Anything else → exit 7 at once: a 404 or 403, or a certificate failure (60), which looks like interception and does not heal.
• tools/net.sh gh <args> does the same for read-only gh calls, classifying stderr by the texts measured above. The stderr of a definite failure is passed through verbatim, because check-updates.sh compares release not found exactly. gh pr create and gh workflow run are not wrapped: a create that timed out after GitHub accepted it would be repeated.

Callers:

place	before	after
fetch.sh get (artifacts, .sig, manifest)	curl --retry 5 --retry-all-errors, exit 22 for everything, 404 re-asked	net.sh: outage → 8, 404 → 7
fetch.sh checksum / manifest size mismatch	exit 1	exit 7
fetch.sh unpinned source archive	any failure → "no source archive", recorded as sourceless, exit 0	404 still does that; an outage exits 8 instead of recording a copyleft package as sourceless
check-updates.sh gh release view, gh api …/releases, gh release download	exit 1, not retried; the message only guessed "an outage clears on a later run"	retried; package stops with 8; the run exits 8 only when every stop was an outage, 1 otherwise
land-updates.sh gh pr list, gh api …/check-runs, gh api …/matching-refs	exit 1, not retried	retried; run exits 8 when only outages stopped branches, 1 otherwise
intake-check.sh gh release view, manifest curl	an outage posted "No release … or the repository is private" or "Could not fetch the manifest" to the requester	exit 8; intake.yml posts "GitHub or the manifest's host did not answer… you do not need to change anything"
update.yml publish-dispatch job (gh api, gh run list, gh workflow run)	unchanged	unchanged: hourly schedule is the retry, and the dispatch is a write

No automatic step-level rerun on exit 8. pre-build runs inside owfeed's reusable feed.yml, and the loop it runs is not idempotent: fetch.sh appends to dist/staged.txt and dist/sources/staged.txt, so re-running the step after a partial fetch would duplicate rows that check-tree.sh and sources.sh read. The in-place retries cover a CDN blip; an incident longer than 75 s ends as exit 8 with an annotation that says to rerun. owfeed/owfeed#21 makes owfeed's own CLI consistent with the same split.

Tests

New tools/test-net.sh (stub curl and gh on PATH, no network), added to the tools job in pr.yml. It covers net.sh get (200, 404, 500 throughout, 503 then 200, 429, 502, exits 6/7/28/56, certificate 60, the Actions annotation), net.sh gh (release not found verbatim, HTTP 503, cannot connect, HTTP 404, 502 then an answer) and fetch.sh end to end (artifact 500/404, checksum mismatch, source archive 404 vs 503).

test-check-updates.sh: b0-outage is asked 5 times and reported as an outage, b1-missing once. The fifth run has only outages (a 503 on download plus the 502 on view) and asserts exit 8 with nothing pushed. test-land-updates.sh: gh pr list 502 is asked 5 times, the run exits 8, and a failed git step exits 1.

Fails before, passes after. The same tests against the origin/main scripts (FETCH=, CHECK_UPDATES=, LAND_UPDATES= overrides):

FAIL artifact 500 throughout: exited 22, expected 8
FAIL artifact 500 throughout: asked 1 time(s), expected 5
FAIL artifact 404: exited 22, expected 7
FAIL checksum mismatch: exited 1, expected 7
FAIL source 503 throughout: exited 0, expected 8
FAIL source 503 throughout: recorded as sourceless: example 1.0.0-r1 example https://github.com/example/example/archive/refs/tags/v1.0.0.tar.gz
FAIL asked for the latest release of example/b0-outage 1 time(s), expected 5
FAIL exited 1, expected 8: only GitHub failed
FAIL the 503 download was attempted 1 time(s), expected 5
FAIL exited 1, expected 8: every failure was GitHub answering 502
FAIL gh pr list was asked 1 time(s) for gamma; tools/net.sh makes 5 attempts

With this branch (and the fixture fix in eb04986), all four tools/test-*.sh print PASS, and dash -n tools/*.sh is clean.

RUNBOOK (EN/RU), under "A check is red": !! upstream outage / exit 8 → rerun (and what the old curl: (22) … 500 ×6 / exit 22 meant); curl: (22) … 404 / exit 7 → wrong tag or file name.