fix: retry a failing download server and say it was an outage - #36
Merged
Merged
Conversation
Release listings, the VM image and its sha256sums, and out-of-feed packages are retried on a 5xx, 429 or dropped connection; a 404 is asked once. A failure that outlasts the retries says it is an outage to wait out, not a problem with owlab.yaml. setup retries gh release reads while GitHub is failing and exits 8 when that outlasts 4 attempts; an attestation API or Sigstore that cannot be reached is reported as an outage, not as a binary that does not verify.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
owlab fetches from servers it does not run: release listings (
owlab releases, release context), the VM image and itssha256sums, and out-of-feed packages (compose.Fetch). Every one of those used a barehttp.Client. One 502 from a mirror ended the command withGET <url>: 502 Bad Gateway, the same kind of message as a release that does not exist, and nothing was asked twice.setup/install.shhad the same gap withgh: one 5xx from the release API failed the step, and an attestation API or Sigstore that could not be reached printed "does not verify … refusing to install it".Companion to owfeed/owfeed-packages#73 and owfeed/owfeed#21, which make owfeed's 7-vs-8 exit contract hold across the ecosystem.
Design
No exit-code distinction in the owlab binary. Nothing reads owlab's exit code beyond zero/non-zero: owfeed's
crosscheck.ymlfails on any non-zero, and owlab passes a container's exit code through (dockercli.ExitCode), so claiming 8 would collide with a real container exit. Per the task, owlab gets retries and messages that say which kind of failure it was.internal/netxis a copy of owfeed's package of the same name. The repositories share no Go module (owfeeddocs/ECOSYSTEM.md).Transportretries GET/HEAD on 408/425/429/5xx and on transient transport errors (*net.OpError,*net.DNSError, timeouts, reset, unexpected EOF): 4 attempts, 2/4/8 s apart. A 404 is asked once. Certificate errors are not retried. Writes pass through.StatusErrorkeepsGET <url>: <status>and appends(the server is failing, not refusing: owlab retried; run the command again later)for transient statuses.OutageError:unreachable after 4 attempts (a network or server outage, not a problem with the config; run the command again later): <err>.Wired into
internal/upstream(get,head),internal/qemu/image.go(expectedSum,fetchTo; the existing "not found … check the release number" message for 404 is kept) andinternal/compose/extra.go(Fetch).HasContainerImageis left alone: any failure there already falls back safely to "the image exists".setup/install.sh:gh_readwrapsgh release view/gh release download. It classifies gh's stderr (5xx/429,error connecting to,proxyconnect,i/o timeout, …) as an outage and retries 4 times, 5/10/20 s apart. When that is exhausted it prints::error title=Upstream outage (safe to rerun)::and exits 8. An answer (release not found,HTTP 404) exits 7 at once.gh attestation verifyoutput matching the same patterns, pluserror creating Sigstore verifier, is retried and then reported as "not checked, rerun", exit 8. A real verification failure is still exit 1 with the existing refusal.Measurements (gh 2.99.0)
gh exits 1 for all of them and makes exactly one request per call.
setup/install.shagainst a stubghon PATH (not committed):Tests
internal/netx/netx_test.go: the classification table; 502, 503, 200 retried to 200; 404 asked once; persistent 503 message; closed port givesOutageErrorwith the outage wording; POST not replayed.internal/qemu/fetch_outage_test.go:fetchToagainst a mirror that answers 502 once and then the image succeeds in 2 requests; a 404 is asked once and keeps the "not found" message. Onmainthe first case fails on the 502.go build ./...,go vet ./...,gofmt -l .(empty),GOOS=windows go build ./cmd/owlabandbash -n setup/install.shall pass.go test ./...passes exceptTestMissingQEMUSaysWhereItLookedAndWhatToRun, which fails locally only because Homebrew's qemu is on PATH (unrelated, and passes on CI).Docs
docs/troubleshooting.md/_ru.md: a new "A download failed" section by symptom (transient status/unreachable → run again later; 404 → check release/target/URL;setupexit 8 vs exit 1).CHANGELOG.md## [Unreleased].