Repository navigation
🐛 fix(ci): retry go mod download in Go workflow jobs on transient proxy resets - #15178
Conversation
…xy resets Add a reusable composite action (.github/actions/go-mod-download-retry) that runs `go mod download` with up to 5 attempts and exponential backoff, and call it right after the Go cache guard in every Go job of v2-ci, v2-tests, go-security-analysis and testutil-guard. The first module-fetching step (golangci-lint, govulncheck, gosec, go build, the sharded go test slices, the coverage scorer's go list) then reads a warm GOMODCACHE instead of racing proxy.golang.org, so a single "stream error: ... INTERNAL_ERROR; received from peer" no longer fails the job. Completes the Docker builder-stage retry from the earlier fix for the same tracker; job semantics are otherwise unchanged. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Decision needed: @clubanderson as the OWNERS approver, confirm this workflow-file change (14 call sites of a new local composite action, no permission or secret changes) is cleared to merge; it is routed only because it touches .github/workflows/*.
Looks correct to me. Read .github/actions/go-mod-download-retry/action.yml in full and every one of the 14 uses: insertions against v5.
Verified:
action.yml:31-51—set -uo pipefailwithout-e,rccaptured per attempt, non-zero exit only on the final attempt (exit "$rc", line 45), so the step is exactly as strict as the unguarded download it replaces. Backoff is 2/4/8/16s;attemptsis validated as a positive integer (line 33).working-directorydefaults tosrc, which holdssrc/go.work;go mod downloadwith no arguments in workspace mode fills the cache for all workspace modules, so latergo build/go test/lint steps read a warmGOMODCACHE.- The step runs after
go-cache-guard.sh, which exportsGOMODCACHEviaGITHUB_ENV, so the retry fills the same cache the later steps use. if:guards: inv2-tests.ymlevery insertion next to asteps.select.outputs.run == 'true'/steps.cov-scope.outputs.score == 'true'step carries the same guard; the two insertions without one (list-packages~line 1129 andwarm-build-cache~line 2709) follow a cache-guard step that is itself unguarded, so no previously skipped lane now runs.- No
permissions:block, secret, or third-party action reference changes anywhere in the diff.
Not a duplicate of #14990 (Docker builder stages); this is the workflow half that PR's body said it could not push.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Looks correct to me. Read the composite action and every insertion site against v5; the retry does what the body says and nothing else.
What I verified at 6368535:
.github/actions/go-mod-download-retry/action.yml:31-51—set -uo pipefailwithout-e, so a failedgo mod downloadlands inrcand the loop continues; final attempt exits with the realrc(:45-47);attemptsis validated as a positive integer (:32-34). Backoff 2/4/8/16s as stated.- Every insertion is directly after
Guard shared Go cache(or aftersetup-goin the coverage scorer,v2-tests.yml:2063), i.e. aftersetup-gohas run, and carries the job's existingif:(steps.select.outputs.run,steps.cov-scope.outputs.score). Nopermissions:block changes in any of the four workflows. src/go.workmakessrc/a two-module workspace;go mod downloadwith no args in workspace mode fetches both the root and./cruxmodules, matching whatsrc/Dockerfile:42already does.
One info note, not a defect: GOFLAGS: -mod=mod (action.yml:28) is a no-op here — go mod download does not define -mod, so the flag is ignored (confirmed with go1.27.3 in a scratch workspace: GOFLAGS=-mod=mod go mod download → rc 0). It is actually good that it's ignored, because in workspace mode Go refuses -mod=mod outright (go: -mod may only be set to readonly or vendor when in workspace mode), so if that env line is ever copied onto a go build/go test step under src/ it will fail every run. Consider dropping the line or commenting why it's there; it's scoped to the composite step so it cannot leak today.
Body matches the diff; changelog fragment name follows the fixed-<pr>-slug.md convention.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 6368535. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
What
Adds a reusable composite action,
.github/actions/go-mod-download-retry, that runsgo mod download(withGOFLAGS=-mod=mod) up to 5 times with exponential backoff (2s, 4s, 8s, 16s), and calls it right after the "Guard shared Go cache" step in every Go job of:.github/workflows/v2-ci.yml—build-and-test.github/workflows/v2-tests.yml—test-hub,test-sliced,test-agent,list-packages,test-rest, the dashboard test lane, the coverage scorer andwarm-build-cache.github/workflows/go-security-analysis.yml—govulncheck,gosec,golangci-lint, NOTICE regeneration.github/workflows/testutil-guard.ymlEach step inherits the job's existing
if:guard, so skipped lanes stay skipped. A changelog fragment is included.Why
The tracker records six PRs on five distinct checks (golangci-lint,
test (… shuffle/dashboard …),coverage,build-and-test) failing on a singlestream error: stream ID N; INTERNAL_ERROR; received from peerfrom proxy.golang.org, with the base branch green on the same checks. #14990 added the retry to the Docker builder stages (src/Dockerfile,src/Dockerfile.contributor,.oss-scanner/Dockerfile), which also coversoverlayfs-exec-guard(it buildssrc/Dockerfile), but could not touch workflow files because the hive's App token lacks theworkflowspermission. This PR lands the remaining workflow-side guard: the first module-fetching step in each job now reads a warmGOMODCACHEinstead of racing the proxy. Go still verifies every module againstgo.sum, so a retry can never substitute a different artifact; a still-failing fifth attempt fails the step exactly like the unguarded download did.Validated with
actionlinton the four workflows (no new findings; the pre-existing shellcheck warnings inoverlayfs-exec-guardare unchanged).Follows up on #14990.
Fixes #14976