Skip to content

fix: retry a failing download server and say it was an outage - #36

Merged
VizzleTF merged 1 commit into
mainfrom
fix/outage-vs-failure
Sep 14, 2026
Merged

VizzleTF merged 1 commit into
mainfrom
fix/outage-vs-failure

Conversation

@VizzleTF

@VizzleTF VizzleTF commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Problem

owlab fetches from servers it does not run: release listings (owlab releases, release context), the VM image and its sha256sums, and out-of-feed packages (compose.Fetch). Every one of those used a bare http.Client. One 502 from a mirror ended the command with GET <url>: 502 Bad Gateway, the same kind of message as a release that does not exist, and nothing was asked twice. setup/install.sh had the same gap with gh: one 5xx from the release API failed the step, and an attestation API or Sigstore that could not be reached printed "does not verify … refusing to install it".

Companion to owfeed/owfeed-packages#73 and owfeed/owfeed#21, which make owfeed's 7-vs-8 exit contract hold across the ecosystem.

Design

No exit-code distinction in the owlab binary. Nothing reads owlab's exit code beyond zero/non-zero: owfeed's crosscheck.yml fails on any non-zero, and owlab passes a container's exit code through (dockercli.ExitCode), so claiming 8 would collide with a real container exit. Per the task, owlab gets retries and messages that say which kind of failure it was.

internal/netx is a copy of owfeed's package of the same name. The repositories share no Go module (owfeed docs/ECOSYSTEM.md).

  • Transport retries GET/HEAD on 408/425/429/5xx and on transient transport errors (*net.OpError, *net.DNSError, timeouts, reset, unexpected EOF): 4 attempts, 2/4/8 s apart. A 404 is asked once. Certificate errors are not retried. Writes pass through.
  • StatusError keeps GET <url>: <status> and appends (the server is failing, not refusing: owlab retried; run the command again later) for transient statuses.
  • OutageError: unreachable after 4 attempts (a network or server outage, not a problem with the config; run the command again later): <err>.

Wired into internal/upstream (get, head), internal/qemu/image.go (expectedSum, fetchTo; the existing "not found … check the release number" message for 404 is kept) and internal/compose/extra.go (Fetch). HasContainerImage is left alone: any failure there already falls back safely to "the image exists".

setup/install.sh:

  • gh_read wraps gh release view / gh release download. It classifies gh's stderr (5xx/429, error connecting to, proxyconnect, i/o timeout, …) as an outage and retries 4 times, 5/10/20 s apart. When that is exhausted it prints ::error title=Upstream outage (safe to rerun):: and exits 8. An answer (release not found, HTTP 404) exits 7 at once.
  • gh attestation verify output matching the same patterns, plus error creating Sigstore verifier, is retried and then reported as "not checked, rerun", exit 8. A real verification failure is still exit 1 with the existing refusal.

Measurements (gh 2.99.0)

release view, 5xx (stub via github.localhost) -> HTTP 503: <message> (http://api.github.localhost/repos/o/r/releases/latest)
gh api, 5xx                                   -> gh: <message> (HTTP 503)
unreachable host                              -> error connecting to nonexistent.invalid check your internet connection or https://githubstatus.com
unreachable proxy                             -> ... proxyconnect tcp: dial tcp 127.0.0.1:1: connect: connection refused
attestation verify, proxy unreachable         -> error creating Sigstore verifier: no valid Sigstore verifiers could be initialized
attestation verify, unattested file           -> Error: HTTP 404: Not Found (https://api.github.com/repos/owfeed/owfeed/attestations/sha256:…)
release view / download, missing tag          -> release not found

gh exits 1 for all of them and makes exactly one request per call.

setup/install.sh against a stub gh on PATH (not committed):

download: release not found          exit=7 calls=1  0s
download: HTTP 503                   exit=8 calls=4  35s  (attempts 1..3 logged, then ::error title=Upstream outage)
attestation: HTTP 404                exit=1          "does not verify ... refusing to install it"
attestation: Sigstore unreachable    exit=8 calls=4  36s  "not found to be wrong; it was not checked. Rerun the job."

Tests

  • internal/netx/netx_test.go: the classification table; 502, 503, 200 retried to 200; 404 asked once; persistent 503 message; closed port gives OutageError with the outage wording; POST not replayed.
  • internal/qemu/fetch_outage_test.go: fetchTo against a mirror that answers 502 once and then the image succeeds in 2 requests; a 404 is asked once and keeps the "not found" message. On main the first case fails on the 502.

go build ./..., go vet ./..., gofmt -l . (empty), GOOS=windows go build ./cmd/owlab and bash -n setup/install.sh all pass. go test ./... passes except TestMissingQEMUSaysWhereItLookedAndWhatToRun, which fails locally only because Homebrew's qemu is on PATH (unrelated, and passes on CI).

Docs

docs/troubleshooting.md / _ru.md: a new "A download failed" section by symptom (transient status/unreachable → run again later; 404 → check release/target/URL; setup exit 8 vs exit 1). CHANGELOG.md ## [Unreleased].

Release listings, the VM image and its sha256sums, and out-of-feed
packages are retried on a 5xx, 429 or dropped connection; a 404 is
asked once. A failure that outlasts the retries says it is an outage
to wait out, not a problem with owlab.yaml.

setup retries gh release reads while GitHub is failing and exits 8
when that outlasts 4 attempts; an attestation API or Sigstore that
cannot be reached is reported as an outage, not as a binary that does
not verify.
@VizzleTF
VizzleTF merged commit b794fa1 into main Sep 14, 2026
15 checks passed
@VizzleTF
VizzleTF deleted the fix/outage-vs-failure branch September 14, 2026 12:08
VizzleTF added a commit that referenced this pull request Sep 14, 2026
Retry a failing download server and GitHub reads in setup/install.sh, and say when a failure was an outage (#36); example and doc names made generic (#35). Move action/setup references and SECURITY.md's download example to v0.6.5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant