Repository navigation
π fix(ci): bound gosec concurrency so Scan stops losing its runner - #16045
Conversation
|
@Karibhou this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Your relay session received this as a |
gosec v2.29.0 defaults -concurrency to runtime.NumCPU(). Inside a self-hosted runner pod that is the node's 32 vCPUs, not the pod's 6-CPU limit, and every worker type-checks and builds SSA for its own package. Thirty-two of those at once push the pod far past its 3Gi request on an overcommitted node, and it is killed mid-Scan: "The self-hosted runner lost communication with the server", logs BlobNotFound. Evidence: in the four failing jobs cited by the issue (124530335184, 124538397043, 124527124413, 124542607459) the job died in the Scan step every time, on four different runner pods, while the green runs finish Scan in about four minutes. Pin -concurrency 4 (at or below the runner CPU limit), regenerate the integration twin, and add a workflow contract asserting both gosec invocations pass an explicit -concurrency no higher than the runner container CPU limit in hive-runners-lke-values.yaml. Fixes hivecommons#16040 Hive-Run: hivecommons#16040 Hive-Plan: issue-16040 asks 1-2 (runner-side evidence, gosec resource profile) Hive-Spec: issue-16040#ask-2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRH6kn7bGF1abH5x72U7zL Signed-off-by: Karibhou <157208370+Karibhou@users.noreply.github.com>
β¦ency pin hivecommons#16035 composed the gosec Scan job into integration-candidate.yml, so after rebasing onto it the generator --check (build-and-test, "Check readonly candidate CI generation") flagged the candidate as stale. Rerun generate-integration-ci.py so the composed Scan step carries -concurrency 4 too. Refs hivecommons#16040 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Karibhou <157208370+Karibhou@users.noreply.github.com>
4830cb1 to
20eb629
Compare
20eb629
|
|
The |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Decision needed: @clubanderson should confirm the three workflow edits (gosec -concurrency 4 in go-security-analysis.yml, integration-technical-go-security-analysis.yml, integration-candidate.yml) are intended before the sentinel-alert label comes off.
The code change is a single flag on three Scan steps and I found no defect in it. I am routing rather than clearing it only because the PR carries sentinel-alert for touching .github/workflows/*, which is a maintainer's call by policy.
What I verified at head 20eb629 (checked out and run locally):
.github/workflows/go-security-analysis.yml:265βgosec -concurrency 4 -exclude-generated ./... ./crux/...; gosec is pinned atv2.29.0(:236), which accepts-concurrency.continue-on-error: trueis untouched, so the step's gating behaviour is unchanged.integration-technical-go-security-analysis.yml:122andintegration-candidate.yml:1859carry the identical line;python3 .github/scripts/generate-integration-ci.py --checkexits 0, so the generated files are in sync with the source.src/deploy/ci-runners/lke/hive-runners-lke-values.yaml:168β therunnercontainer's CPU limit is"6", so4is inside it; the new contract test reads that value rather than hard-coding it.python3 .github/scripts/test-native-queue-workflows.pyβ 5 tests OK on this head.- No permissions, triggers,
runs-on, or secrets blocks change in any of the three workflows β the diff is the onerun:line plus a comment in each.
Open question, not a finding: the root-cause story (32 NumCPU workers each running packages.Load β pod evicted) is plausible and consistent with the Scan-step deaths cited, but it is not verifiable from the diff or the tree, and the memory limit on that container is 16Gi (:169), not the 3Gi request the body cites, so eviction under node pressure is the likelier mechanism than a cgroup OOM. Either way -concurrency 4 is a safe bound β worst case Scan runs longer β and the PR body already names GOMEMLIMIT as the follow-up if runner loss persists. Fork PRs run gosec on ubuntu-latest, so this PR's own green Scan does not exercise the self-hosted path.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Routing to a human because this PR carries What actually changes β read on head
Mechanism is sound β gosec's Also flagged by the hive: label Decision for @clubanderson: this is a one-flag, non-privilege change to three workflows from a fork β if that is acceptable, it looks correct to me. |
|
Needs a decision, not a merge:
Head: verdict requires_human @20eb629, 60/60 checks green, DCO yes, sentinel alert, soak 10m0s elapsed. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Needs a decision, not a merge:
Head: verdict requires_human @20eb629, 60/60 checks green, DCO yes, sentinel alert, soak 10m0s elapsed. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Follow-up on the reviewer note at β hive: backend=claude model=claude-opus-5-5 |
clubanderson
left a comment
There was a problem hiding this comment.
Maintainer decision, requested by the owner: approve the workflow edits at 20eb629, including their arrival from this fork. The diff bounds gosec at four workers in all three workflow Scan steps and adds a structural regression check. It does not change permissions, secrets, triggers, runner selection, scan targets or gating behavior. All eleven required contexts are green on this head. The sentinel workflow-change concern is resolved for this exact diff; a future head still requires ordinary reevaluation. Runner loss is consistent with resource pressure but is not conclusive OOM evidence. Land this small reliability mitigation before the overlapping #16026, then verify the real self-hosted path under incident #16048. Fork CI does not establish that production runner recovery is complete. Extending the structural test to integration-candidate.yml is a suitable focused follow-up; generator synchronization already covers the generated file. Do not remove the genuine overlap marker without reconciliation or bypass any remaining merge gate.
|
/lgtm Owner-requested decision recorded as a native APPROVED review on head 20eb629, and the reviewed workflow-change sentinel label is cleared. Merge #16045 before #16026; preserve the bounded scanner concurrency when reconciling that later branch. Keep the existing overlap marker until reconciled, normal CI/merge gates, and the self-hosted recovery verification in #16048. This decision does not certify OOM as the observed cause or claim the production incident is already fixed. |
|
LGTM label has been added. DetailsGit tree hash: e64ae41a0524f8cbef0dec6aeab4a7240ccec459 |
|
@clubanderson approved head |
|
@clubanderson approved head |
97fad87
|
|
Needs a decision, not a merge:
Head: verdict approved @97fad87, 49/54 checks green, 5 pending, DCO yes, sentinel alert, soak 0s of 10m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
/lgtm At the ownerβs explicit November 2 request to get this moving, reviewed current head 97fad87 and am clearing the sensitive-workflow Sentinel label for this exact diff. This is the intended four-worker gosec cap in the source workflow and both generated workflows, plus its structural regression check. No permissions, secrets, triggers, runner selection, scan targets, or gating changes. A current-head maintainer APPROVED review is already present. Preserve the real overlap marker and all ordinary CI/soak/review requirements; a later push must receive fresh evaluation. Resource pressure is a plausible explanation, not proven OOM attribution; actual self-hosted recovery remains to be demonstrated after merge. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
LGTM label has been added. DetailsGit tree hash: e8dc5f454310ab6144a8401e0aebd742767881ea |
Merge current v5, preserving #16045 gosec concurrency 4 and its generated integration workflows, #16027 preparation rail, and existing readiness regressions. Prior-head gosec job 125331848206 explicitly reported lost self-hosted runner communication; refresh exact-head CI without weakening any gate or removing hold. Signed-off-by: Andy Anderson <andy@clubanderson.com>
Summary
gosec (insecure patterns)keeps dying with "The self-hosted runner lost communication with the server" (logsBlobNotFound). This bounds gosec's worker count so the Scan step no longer overruns the runner pod.Root cause (evidence, not a guess)
Scanstep. Install and every earlier step were green. Each failure ran on a different runner pod (β¦-zfqqh,β¦-4kvkv,β¦-bqp9t,β¦-tkphn), so this is not one bad host.-concurrencytoruntime.NumCPU()(cmd/gosec/main.go). Inside a pod, that returns the node's 32 vCPUs, not the pod's 6-CPU limit. Go 1.25 makesGOMAXPROCScgroup-aware, but notNumCPU.packages.Load(types and syntax) and builds SSA (analyzer.go,Process). With 32 workers in parallel, the pod goes far past its3Girequest (src/deploy/ci-runners/lke/hive-runners-lke-values.yaml). The nodes are deliberately overcommitted, so this pod is the first one evicted or OOM-killed, which looks exactly like "runner lost".Change
go-security-analysis.yml:gosec -concurrency 4 β¦, with a comment explaining why.integration-technical-go-security-analysis.yml: regenerated withgenerate-integration-ci.py.--checkpasses.test-native-queue-workflows.py: a new contract test. Both gosec invocations must pass an explicit-concurrencybetween 1 and the runner container's CPU limit, which the test reads from the LKE values file. It failed before the fix and passes after it.What the issue asked for that already exists
ci-infra-rerun.ymlalready watches Go Security Analysis.ci-infra-signatures.tsvalready classifies this exact annotation asrunner-lost, and failed infra jobs are rerun once. That is why π± chore(rfcnotify): add tests for card.go helper functionsΒ #16014 got an attempt 2, which then died the same way. A retry cannot cure a deterministic overload, so this PR fixes the load itself.Test plan
python3 .github/scripts/test-native-queue-workflows.py: 5/5. The new test failed before the fix.python3 .github/scripts/generate-integration-ci.py --check: OK.python3 .github/scripts/test-ci-infra.py: 37/37.gosecScan step on same-repo PRs. A fork PR like this one runs gosec onubuntu-latest, so this PR's CI does not exercise the self-hosted path. Expect Scan to take somewhat longer than 4 min, well inside the 25-min timeout.GOMEMLIMITon the Scan step, or capture peak RSS with/usr/bin/time -v. Both are left out on purpose to keep this to one change.Fixes #16040
π€ Generated with Claude Code
https://claude.ai/code/session_01HRH6kn7bGF1abH5x72U7zL
β hive: backend=claude model=claude-opus-5-5