Skip to content

🐛 fix(ci): retry go mod download in Go workflow jobs on transient proxy resets - #15178

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
fix/ci-go-proxy-retry-workflows
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
fix/ci-go-proxy-retry-workflows

Conversation

@clubanderson

Copy link
Copy Markdown
Member

What

Adds a reusable composite action, .github/actions/go-mod-download-retry, that runs go mod download (with GOFLAGS=-mod=mod) up to 5 times with exponential backoff (2s, 4s, 8s, 16s), and calls it right after the "Guard shared Go cache" step in every Go job of:

  • .github/workflows/v2-ci.yml — build-and-test
  • .github/workflows/v2-tests.yml — test-hub, test-sliced, test-agent, list-packages, test-rest, the dashboard test lane, the coverage scorer and warm-build-cache
  • .github/workflows/go-security-analysis.yml — govulncheck, gosec, golangci-lint, NOTICE regeneration
  • .github/workflows/testutil-guard.yml

Each step inherits the job's existing if: guard, so skipped lanes stay skipped. A changelog fragment is included.

Why

The tracker records six PRs on five distinct checks (golangci-lint, test (… shuffle/dashboard …), coverage, build-and-test) failing on a single stream error: stream ID N; INTERNAL_ERROR; received from peer from proxy.golang.org, with the base branch green on the same checks. #14990 added the retry to the Docker builder stages (src/Dockerfile, src/Dockerfile.contributor, .oss-scanner/Dockerfile), which also covers overlayfs-exec-guard (it builds src/Dockerfile), but could not touch workflow files because the hive's App token lacks the workflows permission. This PR lands the remaining workflow-side guard: the first module-fetching step in each job now reads a warm GOMODCACHE instead of racing the proxy. Go still verifies every module against go.sum, so a retry can never substitute a different artifact; a still-failing fifth attempt fails the step exactly like the unguarded download did.

Validated with actionlint on the four workflows (no new findings; the pre-existing shellcheck warnings in overlayfs-exec-guard are unchanged).

Follows up on #14990.

Fixes #14976

…xy resets

Add a reusable composite action (.github/actions/go-mod-download-retry)
that runs `go mod download` with up to 5 attempts and exponential
backoff, and call it right after the Go cache guard in every Go job of
v2-ci, v2-tests, go-security-analysis and testutil-guard. The first
module-fetching step (golangci-lint, govulncheck, gosec, go build, the
sharded go test slices, the coverage scorer's go list) then reads a warm
GOMODCACHE instead of racing proxy.golang.org, so a single
"stream error: ... INTERNAL_ERROR; received from peer" no longer fails
the job. Completes the Docker builder-stage retry from the earlier
fix for the same tracker; job semantics are otherwise unchanged.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15159 🌱 chore: remove unreachable MITM cancellation guard same files .github/workflows/v2-ci.yml
#14822 🐛 fix: preserve owner intent through PR decision paths same files .github/workflows/v2-ci.yml

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson as the OWNERS approver, confirm this workflow-file change (14 call sites of a new local composite action, no permission or secret changes) is cleared to merge; it is routed only because it touches .github/workflows/*.

Looks correct to me. Read .github/actions/go-mod-download-retry/action.yml in full and every one of the 14 uses: insertions against v5.

Verified:

  • action.yml:31-51 — set -uo pipefail without -e, rc captured per attempt, non-zero exit only on the final attempt (exit "$rc", line 45), so the step is exactly as strict as the unguarded download it replaces. Backoff is 2/4/8/16s; attempts is validated as a positive integer (line 33).
  • working-directory defaults to src, which holds src/go.work; go mod download with no arguments in workspace mode fills the cache for all workspace modules, so later go build/go test/lint steps read a warm GOMODCACHE.
  • The step runs after go-cache-guard.sh, which exports GOMODCACHE via GITHUB_ENV, so the retry fills the same cache the later steps use.
  • if: guards: in v2-tests.yml every insertion next to a steps.select.outputs.run == 'true' / steps.cov-scope.outputs.score == 'true' step carries the same guard; the two insertions without one (list-packages ~line 1129 and warm-build-cache ~line 2709) follow a cache-guard step that is itself unguarded, so no previously skipped lane now runs.
  • No permissions: block, secret, or third-party action reference changes anywhere in the diff.

Not a duplicate of #14990 (Docker builder stages); this is the workflow half that PR's body said it could not push.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. Read the composite action and every insertion site against v5; the retry does what the body says and nothing else.

What I verified at 6368535:

  • .github/actions/go-mod-download-retry/action.yml:31-51 — set -uo pipefail without -e, so a failed go mod download lands in rc and the loop continues; final attempt exits with the real rc (:45-47); attempts is validated as a positive integer (:32-34). Backoff 2/4/8/16s as stated.
  • Every insertion is directly after Guard shared Go cache (or after setup-go in the coverage scorer, v2-tests.yml:2063), i.e. after setup-go has run, and carries the job's existing if: (steps.select.outputs.run, steps.cov-scope.outputs.score). No permissions: block changes in any of the four workflows.
  • src/go.work makes src/ a two-module workspace; go mod download with no args in workspace mode fetches both the root and ./crux modules, matching what src/Dockerfile:42 already does.

One info note, not a defect: GOFLAGS: -mod=mod (action.yml:28) is a no-op here — go mod download does not define -mod, so the flag is ignored (confirmed with go1.27.3 in a scratch workspace: GOFLAGS=-mod=mod go mod download → rc 0). It is actually good that it's ignored, because in workspace mode Go refuses -mod=mod outright (go: -mod may only be set to readonly or vendor when in workspace mode), so if that env line is ever copied onto a go build/go test step under src/ it will fail every run. Consider dropping the line or commenting why it's there; it's scoped to the composite step so it cannot leak today.

Body matches the diff; changelog fragment name follows the fixed-<pr>-slug.md convention.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 1aa9404 into v5 Oct 29, 2026
81 of 85 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/ci-go-proxy-retry-workflows branch October 29, 2026 00:12
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#151786368535` sha256:840cb9ed6c2d79b37f260332936b3766687f40504dd79d68638cb99bbcbf990f — download from the dashboard or `hivectl review evidence #15178`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 6368535.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[shared-ci] Go module proxy stream resets (INTERNAL_ERROR) failing golangci-lint, overlayfs-exec-guard and docker (hub) across hivecommons/hive

1 participant