Skip to content

πŸ› fix(ci): bound gosec concurrency so Scan stops losing its runner - #16045

Merged
kubestellar-prow[bot] merged 3 commits into
hivecommons:v5from
Karibhou:ci/16040-gosec-runner-loss
Nov 2, 2026
Merged

kubestellar-prow[bot] merged 3 commits into
hivecommons:v5from
Karibhou:ci/16040-gosec-runner-loss

Conversation

@Karibhou

Copy link
Copy Markdown
Contributor

Summary

gosec (insecure patterns) keeps dying with "The self-hosted runner lost communication with the server" (logs BlobNotFound). This bounds gosec's worker count so the Scan step no longer overruns the runner pod.

Root cause (evidence, not a guess)

  • Where it dies: in all four failing jobs cited in the issue (124530335184, 124538397043, 124527124413, 124542607459) the job died in the Scan step. Install and every earlier step were green. Each failure ran on a different runner pod (…-zfqqh, …-4kvkv, …-bqp9t, …-tkphn), so this is not one bad host.
  • Green baseline: on v5 runs 41202519218 and 41199008359, Scan took about 4 minutes.
  • Why gosec specifically: gosec v2.29.0 defaults -concurrency to runtime.NumCPU() (cmd/gosec/main.go). Inside a pod, that returns the node's 32 vCPUs, not the pod's 6-CPU limit. Go 1.25 makes GOMAXPROCS cgroup-aware, but not NumCPU.
  • What each worker does: each one runs its own packages.Load (types and syntax) and builds SSA (analyzer.go, Process). With 32 workers in parallel, the pod goes far past its 3Gi request (src/deploy/ci-runners/lke/hive-runners-lke-values.yaml). The nodes are deliberately overcommitted, so this pod is the first one evicted or OOM-killed, which looks exactly like "runner lost".
  • Why the other jobs survive: govulncheck and golangci-lint in the same run do one analysis pass and stay green.

Change

  • go-security-analysis.yml: gosec -concurrency 4 …, with a comment explaining why.
  • integration-technical-go-security-analysis.yml: regenerated with generate-integration-ci.py. --check passes.
  • test-native-queue-workflows.py: a new contract test. Both gosec invocations must pass an explicit -concurrency between 1 and the runner container's CPU limit, which the test reads from the LKE values file. It failed before the fix and passes after it.

What the issue asked for that already exists

  • Asks 2 (retry) and 3 (infra classification): ci-infra-rerun.yml already watches Go Security Analysis. ci-infra-signatures.tsv already classifies this exact annotation as runner-lost, and failed infra jobs are rerun once. That is why 🌱 chore(rfcnotify): add tests for card.go helper functionsΒ #16014 got an attempt 2, which then died the same way. A retry cannot cure a deterministic overload, so this PR fixes the load itself.
  • Ask 1 (runner-side evidence): I have no cluster access. The step and pod evidence above is what is visible from the Actions API.

Test plan

  • python3 .github/scripts/test-native-queue-workflows.py: 5/5. The new test failed before the fix.
  • python3 .github/scripts/generate-integration-ci.py --check: OK.
  • python3 .github/scripts/test-ci-infra.py: 37/37.
  • After merge, watch the self-hosted gosec Scan step on same-repo PRs. A fork PR like this one runs gosec on ubuntu-latest, so this PR's CI does not exercise the self-hosted path. Expect Scan to take somewhat longer than 4 min, well inside the 25-min timeout.
  • Possible follow-up if a runner loss still happens: add GOMEMLIMIT on the Scan step, or capture peak RSS with /usr/bin/time -v. Both are left out on purpose to keep this to one change.

Fixes #16040

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01HRH6kn7bGF1abH5x72U7zL

β€” hive: backend=claude model=claude-opus-5-5

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 31, 2026
@Karibhou
Karibhou requested a review from Danathar as a code owner October 31, 2026 13:17
@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 31, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Karibhou this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • build-and-test

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

Karibhou and others added 2 commits October 31, 2026 14:19
gosec v2.29.0 defaults -concurrency to runtime.NumCPU(). Inside a
self-hosted runner pod that is the node's 32 vCPUs, not the pod's 6-CPU
limit, and every worker type-checks and builds SSA for its own package.
Thirty-two of those at once push the pod far past its 3Gi request on an
overcommitted node, and it is killed mid-Scan: "The self-hosted runner
lost communication with the server", logs BlobNotFound.

Evidence: in the four failing jobs cited by the issue (124530335184,
124538397043, 124527124413, 124542607459) the job died in the Scan step
every time, on four different runner pods, while the green runs finish
Scan in about four minutes.

Pin -concurrency 4 (at or below the runner CPU limit), regenerate the
integration twin, and add a workflow contract asserting both gosec
invocations pass an explicit -concurrency no higher than the runner
container CPU limit in hive-runners-lke-values.yaml.

Fixes hivecommons#16040

Hive-Run: hivecommons#16040
Hive-Plan: issue-16040 asks 1-2 (runner-side evidence, gosec resource profile)
Hive-Spec: issue-16040#ask-2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HRH6kn7bGF1abH5x72U7zL
Signed-off-by: Karibhou <157208370+Karibhou@users.noreply.github.com>
…ency pin

hivecommons#16035 composed the gosec Scan job into integration-candidate.yml, so
after rebasing onto it the generator --check (build-and-test, "Check
readonly candidate CI generation") flagged the candidate as stale. Rerun
generate-integration-ci.py so the composed Scan step carries
-concurrency 4 too.

Refs hivecommons#16040

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Karibhou <157208370+Karibhou@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor
20eb629

⚠️ Sentinel alert β€” maintainer review required

Hive flagged this PR (author @Karibhou, head 20eb62938d0e) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 3 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/go-security-analysis.yml
    • .github/workflows/integration-candidate.yml
    • .github/workflows/integration-technical-go-security-analysis.yml

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@Karibhou

Copy link
Copy Markdown
Contributor Author

The build-and-test failure flagged for head 4830cb1 is resolved on the current head 20eb629: all required checks now pass, with only tide waiting for lgtm/approval. The sentinel alert is expected: this PR's purpose is to change the gosec workflows (bounding gosec concurrency so the Scan job stops losing its runner, #16040), so a maintainer needs to review the three workflow files before the sentinel-alert label is removed.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16026 🌱 ci: reduce PR critical path without dropping checks CONFLICT .github/workflows/go-security-analysis.yml
.github/workflows/integration-candidate.yml
.github/workflows/integration-technical-go-security-analysis.yml

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should confirm the three workflow edits (gosec -concurrency 4 in go-security-analysis.yml, integration-technical-go-security-analysis.yml, integration-candidate.yml) are intended before the sentinel-alert label comes off.

The code change is a single flag on three Scan steps and I found no defect in it. I am routing rather than clearing it only because the PR carries sentinel-alert for touching .github/workflows/*, which is a maintainer's call by policy.

What I verified at head 20eb629 (checked out and run locally):

  • .github/workflows/go-security-analysis.yml:265 β€” gosec -concurrency 4 -exclude-generated ./... ./crux/...; gosec is pinned at v2.29.0 (:236), which accepts -concurrency. continue-on-error: true is untouched, so the step's gating behaviour is unchanged.
  • integration-technical-go-security-analysis.yml:122 and integration-candidate.yml:1859 carry the identical line; python3 .github/scripts/generate-integration-ci.py --check exits 0, so the generated files are in sync with the source.
  • src/deploy/ci-runners/lke/hive-runners-lke-values.yaml:168 β€” the runner container's CPU limit is "6", so 4 is inside it; the new contract test reads that value rather than hard-coding it. python3 .github/scripts/test-native-queue-workflows.py β†’ 5 tests OK on this head.
  • No permissions, triggers, runs-on, or secrets blocks change in any of the three workflows β€” the diff is the one run: line plus a comment in each.

Open question, not a finding: the root-cause story (32 NumCPU workers each running packages.Load β†’ pod evicted) is plausible and consistent with the Scan-step deaths cited, but it is not verifiable from the diff or the tree, and the memory limit on that container is 16Gi (:169), not the 3Gi request the body cites, so eviction under node pressure is the likelier mechanism than a cgroup OOM. Either way -concurrency 4 is a safe bound β€” worst case Scan runs longer β€” and the PR body already names GOMEMLIMIT as the follow-up if runner loss persists. Fork PRs run gosec on ubuntu-latest, so this PR's own green Scan does not exercise the self-hosted path.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Routing to a human because this PR carries sentinel-alert and edits .github/workflows/** (CODEOWNERS: @clubanderson @hanthor @Danathar). I have not approved anything; here is what I verified so a reviewer can decide quickly.

What actually changes β€” read on head 20eb629 against v5:

  • .github/workflows/go-security-analysis.yml:264, integration-technical-go-security-analysis.yml:122, integration-candidate.yml:1859 β€” the only edit in each workflow is gosec -exclude-generated … β†’ gosec -concurrency 4 -exclude-generated …. No permissions: block, action pin, trigger, step or continue-on-error value changes. The Scan step remains continue-on-error: true.
  • .github/scripts/test-native-queue-workflows.py:73-95 β€” new structural test reads limits.cpu of the runner container from src/deploy/ci-runners/lke/hive-runners-lke-values.yaml and asserts both security workflows pass an explicit -concurrency ≀ that limit. I ran python3 .github/scripts/test-native-queue-workflows.py on the PR head: 5 tests, OK. Note it covers the two security workflows but not integration-candidate.yml, which was changed by hand to match.

Mechanism is sound β€” gosec's -concurrency defaults to runtime.NumCPU(), which inside a cgroup-limited pod reports the node's CPUs; bounding it only reduces parallelism in a step whose result is already non-blocking. The runner requests are cpu: 1500m / memory: 3Gi (values.yaml:163-165), so the memory argument in the body is plausible; I did not reproduce the OOM.

Also flagged by the hive: label conflicts-with-open-pr. I cannot enumerate open PRs from here, so a maintainer should check whether another PR edits the same Scan lines before merging.

Decision for @clubanderson: this is a one-flag, non-privilege change to three workflows from a fork β€” if that is acceptable, it looks correct to me.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @20eb629
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @clubanderson should confirm the three workflow edits (gosec -concurrency 4 in go-security-analysis.yml, integration-technical-go-security-analysis.yml, integration-candidate.yml) are intended before the sentinel-alert label comes off.
  • a sentinel alert is open on this PR
  • the automerge lane will not merge this unattended

Head: verdict requires_human @20eb629, 60/60 checks green, DCO yes, sentinel alert, soak 10m0s elapsed.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed. An ordinary contributor cannot release their own PR.
  • Resolve the sentinel alert and have a maintainer clear its label.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @20eb629
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @clubanderson should decide whether this change to .github/workflows (adding -concurrency 4 to the gosec Scan step in go-security-analysis.yml, integration-technical-go-security-analysis.yml and integration-candidate.yml) may land from a fork under the sentinel-alert label.
  • a sentinel alert is open on this PR
  • the automerge lane will not merge this unattended

Head: verdict requires_human @20eb629, 60/60 checks green, DCO yes, sentinel alert, soak 10m0s elapsed.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed. An ordinary contributor cannot release their own PR.
  • Resolve the sentinel alert and have a maintainer clear its label.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@Karibhou

Copy link
Copy Markdown
Contributor Author

Follow-up on the reviewer note at 20eb629: .github/scripts/test-native-queue-workflows.py only checks the two security workflows. The -concurrency 4 in integration-candidate.yml was matched by hand. I am not pushing a new head while @clubanderson's decision is pending, because that would re-run the review and the sentinel. If the change is accepted, I can extend the structural test to cover integration-candidate.yml as well (here or in a follow-up, whichever the maintainer prefers). The conflicts-with-open-pr label is for #16026, which edits the same Scan lines: whichever merges second will need a rebase, and I will repair this branch if it is the one.

β€” hive: backend=claude model=claude-opus-5-5

@clubanderson clubanderson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer decision, requested by the owner: approve the workflow edits at 20eb629, including their arrival from this fork. The diff bounds gosec at four workers in all three workflow Scan steps and adds a structural regression check. It does not change permissions, secrets, triggers, runner selection, scan targets or gating behavior. All eleven required contexts are green on this head. The sentinel workflow-change concern is resolved for this exact diff; a future head still requires ordinary reevaluation. Runner loss is consistent with resource pressure but is not conclusive OOM evidence. Land this small reliability mitigation before the overlapping #16026, then verify the real self-hosted path under incident #16048. Fork CI does not establish that production runner recovery is complete. Extending the structural test to integration-candidate.yml is a suitable focused follow-up; generator synchronization already covers the generated file. Do not remove the genuine overlap marker without reconciliation or bypass any remaining merge gate.

@kubestellar-prow kubestellar-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 2, 2026
@clubanderson

Copy link
Copy Markdown
Member

/lgtm
/approve

Owner-requested decision recorded as a native APPROVED review on head 20eb629, and the reviewed workflow-change sentinel label is cleared. Merge #16045 before #16026; preserve the bounded scanner concurrency when reconciling that later branch. Keep the existing overlap marker until reconciled, normal CI/merge gates, and the self-hosted recovery verification in #16048. This decision does not certify OOM as the observed cause or claim the production incident is already fixed.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: e64ae41a0524f8cbef0dec6aeab4a7240ccec459

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson approved head 20eb62938d0e, which edits CI workflow / action files. That explicit owner approval releases the contributor lane's workflow-path fence: the hive merges this PR itself once the usual gates (CI, soak, holds, review verdict) are clear. A new push re-arms the fence until the new head is approved again.

@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 2, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson approved head 97fad87fa6f7, which edits CI workflow / action files. That explicit owner approval releases the contributor lane's workflow-path fence: the hive merges this PR itself once the usual gates (CI, soak, holds, review verdict) are clear. A new push re-arms the fence until the new head is approved again.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
97fad87

⚠️ Sentinel alert β€” maintainer review required

Hive flagged this PR (author @Karibhou, head 97fad87fa6f7) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 3 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/go-security-analysis.yml
    • .github/workflows/integration-candidate.yml
    • .github/workflows/integration-technical-go-security-analysis.yml

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • a sentinel alert is open on this PR
  • the automerge lane will not merge this unattended

Head: verdict approved @97fad87, 49/54 checks green, 5 pending, DCO yes, sentinel alert, soak 0s of 10m0s.

How to release it:

  • Resolve the sentinel alert and have a maintainer clear its label.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@clubanderson

Copy link
Copy Markdown
Member

/lgtm
/approve

At the owner’s explicit November 2 request to get this moving, reviewed current head 97fad87 and am clearing the sensitive-workflow Sentinel label for this exact diff. This is the intended four-worker gosec cap in the source workflow and both generated workflows, plus its structural regression check. No permissions, secrets, triggers, runner selection, scan targets, or gating changes. A current-head maintainer APPROVED review is already present. Preserve the real overlap marker and all ordinary CI/soak/review requirements; a later push must receive fresh evaluation. Resource pressure is a plausible explanation, not proven OOM attribution; actual self-hosted recovery remains to be demonstrated after merge.

@kubestellar-prow kubestellar-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 2, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: e8dc5f454310ab6144a8401e0aebd742767881ea

@kubestellar-prow
kubestellar-prow Bot merged commit 1e83f1f into hivecommons:v5 Nov 2, 2026
76 of 79 checks passed
clubanderson added a commit that referenced this pull request Nov 2, 2026
Merge current v5, preserving #16045 gosec concurrency 4 and its generated integration workflows, #16027 preparation rail, and existing readiness regressions. Prior-head gosec job 125331848206 explicitly reported lost self-hosted runner communication; refresh exact-head CI without weakening any gate or removing hold.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@Karibhou
Karibhou deleted the ci/16040-gosec-runner-loss branch November 15, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] ci: gosec (insecure patterns) repeatedly loses its self-hosted runner (6+ runs, no code cause)

2 participants