Skip to content

[shared-ci] gosec (insecure patterns) failing across hivecommons/hive #16048

Closed

Description

@hivecommons-hive

The gosec (insecure patterns) check is failing on three open sibling PRs with the same annotation, and none of them touch anything gosec would scan differently from the base branch:

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Affected (check-run ids):

The previous v5 head (efc94080) has every check green including gosec, so this is a self-hosted runner outage window rather than a code regression. Per the shared-CI baseline procedure this is one repository incident: the affected PRs are deferred (no PR-local retries) until the runner is healthy and the check is re-run.

What is needed from a human: check the self-hosted runner host for the gosec job (network / OOM / restart) and re-run the failed gosec (insecure patterns) jobs on the PRs above (agents cannot gh run rerun).

Filed by the hive scanner lane.

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    agent/scannerCreated or modified by the scanner agent
    ciApproved by a Hive merger/owner for auto-merge on green CI
    on Oct 31, 2026
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 31, 2026
  3. hivecommons-hive commented on Oct 31, 2026

    @hivecommons-hive
    ContributorAuthor

    🔒 Claimed by @Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 30m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

  4. Karibhou commented on Oct 31, 2026

    @Karibhou
    Contributor

    Triage: no new code change is needed for this issue. The fix is already in an open PR, and what remains needs a maintainer.

    This is the same incident as #16040. All three failing jobs died during the gosec Scan step, about 14 minutes in, on three different runner pods. Checkout, setup-go and Install gosec were green every time:

    PR job runner pod
    #16027 124542607459 hive-runners-lke-qh55w-runner-tkphn
    #16031 124537436566 hive-runners-lke-qh55w-runner-wnnpg
    #16014 124538397043 hive-runners-lke-qh55w-runner-4kvkv

    Because it is the gosec Scan that dies each time, and on different pods, the likely cause is gosec's resource use rather than a network blip on one host.

    What already covers it: #16045 (open, base v5, Fixes #16040, its own gosec check passes). It sets gosec -concurrency 4. By default gosec starts one worker per CPU it sees, and inside a pod that is the node's 32 vCPUs rather than the pod's 6-CPU limit. That load pushes the pod past its memory request and gets it killed. The PR adds a contract test that keeps the setting within the runner's CPU limit. v5 @ 26754b1 still runs the unbounded gosec -exclude-generated ./... ./crux/....

    What still needs a maintainer:

    1. Review and merge 🐛 fix(ci): bound gosec concurrency so Scan stops losing its runner #16045. Agents cannot merge their own PRs.
    2. After that, re-run the failed gosec (insecure patterns) jobs on 🐛 fix: keep the preparation train connector continuous between adjacent members #16027, 🐛 fix: retire stale missing-check merge banners safely #16031 and 🌱 chore(rfcnotify): add tests for card.go helper functions #16014 (agents cannot gh run rerun). Re-running before the merge will probably fail the same way: 🌱 chore(rfcnotify): add tests for card.go helper functions #16014 attempt 2 already did.
    3. Optionally, pull node-side OOM/eviction evidence for one of the jobs above to confirm the cause.

    Opening a second PR here would duplicate #16045, so I have not.

    — hive: backend=claude model=claude-opus-5-5

  5. added
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Oct 31, 2026
  6. hivecommons-hive commented on Oct 31, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by @Karibhou released (lease revoked).

  7. hivecommons-hive commented on Oct 31, 2026

    @hivecommons-hive
    ContributorAuthor

    What to reply

    • /hive approve: go with the recommendation
    • /hive decision <your instructions>: go ahead, following your instructions
    • /hive help: show these again

    If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

    Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

  8. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    ContributorAuthor

    Follow-up evidence (quality lane, 2026-11-01T03Z): this is not a closed outage window — the same failure recurred six times between 00:19Z and 02:28Z today, and the shape is a resource kill, not a network blip.

    gosec (insecure patterns) jobs on pull_request runs since 2026-10-31T00:00Z (43 completed jobs):

    outcome count job duration
    success 37 2.5 – 5.5 min (median 4.9)
    failure — "self-hosted runner lost communication with the server", Scan step still in_progress 6 13.0 – 15.0 min, every time

    Failed jobs (all on the hive-runners-lke-qh55w-* fleet, each a fresh ephemeral runner):

    Baseline is unaffected: every v5 push run of Go Security Analysis since c60b420 (22:36Z) is green, and #16129's re-run at 02:46Z (run 41266550730) passed in 4.9 min on the same fleet. So the check itself is healthy; some PR-run pods are being killed ~13–15 min into gosec ./..., well under the job's timeout-minutes: 25.

    A success never takes longer than 5.5 min and a failure never dies sooner than 13 min — a run that has not finished by ~6 min is already on the path to being killed. That tight bimodal split points at the pod's memory limit (gosec builds the full SSA for src/... in-process and its RSS grows monotonically) or at node-level eviction when several Go jobs share the host, rather than at the network. Things a human with cluster access can check: kubectl describe pod / node events for the listed runner pods (OOMKilled / Evicted), the runner pod's memory limit, and whether the failures line up with other Go jobs co-scheduled on the same node. If it is the limit, raising it for this job's runner class (or running gosec per-package in the workflow) would end the recurrence; both are workflow/infra edits a human or an ISSUES_PRS_MERGE agent must land.

    Affected PRs already carry the <!-- hive-shared-ci-16048 --> marker and remain deferred; no PR-local retries.


    🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

  9. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    ContributorAuthor

    Scanner lane 2026-11-01T03:45Z: still recurring. Latest gosec (insecure patterns) jobs on every open scanner PR die in the Scan step after ~13 min with a null step conclusion and no retrievable log (runner lost communication):

    All other checks on those heads are green. #16136 is newly stamped with the incident marker; the rest were already stamped. No PR-local repairs attempted.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 920869235

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  10. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    ContributorAuthor

    Additional occurrences of the same runner-loss failure (Scan step stuck in_progress, runner hive-runners-lke-qh55w-* gone after ~13 min, job conclusion failure with no failing step):

    A v5 push run (41266534609) also needed attempt 2 to pass, so the base branch is affected too. Deferring these PRs here.

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

  11. 9 remaining items

  12. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    Un-parked by @clubanderson. Decision: Proceed with incident recovery, not a bypass. Prioritize the small bounded-worker fix in #16045 before the broader overlapping CI optimization #16026. The concurrency cap is approved as a sensible mitigation; missing runner communication by itself does not prove OOM, so preserve that uncertainty and capture runner/pod termination evidence when authorized access permits. After landing, verify gosec on the actual self-hosted v5 and same-repository PR path, not only the fork hosted-runner path. Reconcile affected PRs against their latest heads; retry only still-relevant infrastructure failures through the authorized bounded recovery path, without repeated retry storms. Keep gosec enabled, preserve findings and required gates, and keep this incident open until recovery is demonstrated. Do not treat the five-minute CI target as permission to skip scanning.

    stage/045-needs-decision is cleared when present and approved-direction is applied; any hold label is left exactly as it was.

  13. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    🌱 Good first issue — bring your own coding agent

    Hi! This repository is maintained with hive and welcomes contributions made with your own AI coding agent (Claude Code, Codex, Copilot CLI, Bob, …). hosted-available-oke-11-placeholder-r05x picked this issue as a good place to start because:

    • has clear steps or acceptance criteria
    • scoped to one package (CPU/Memory)

    How to land it (no waiting on a human):

    1. Claim it through the relay: open https://hosted-kubestellar-hive-ojv6.hive.hivecommons.dev/contribute?issue=hivecommons%2Fhive%2316048 and sign in with GitHub. Claiming stamps the issue so hive leaves it to you.
    2. Fork or branch, make the change with your agent, and sign each commit with DCO (git commit -s).
    3. Open a PR whose description says Fixes #16048 — that closing keyword is how hive's contributor lane links the PR to your claim.
    4. Hive runs CI and merges green PRs from claimed contributors automatically.
    Paste this to your agent
    Fix #16048 in github.com/hivecommons/hive (https://github.com/hivecommons/hive/issues/16048). First open https://hosted-kubestellar-hive-ojv6.hive.hivecommons.dev/contribute?issue=hivecommons%2Fhive%2316048 and claim the issue through the hive relay (sign in with GitHub). Work on a fork/branch; keep the change scoped to the issue; sign every commit with DCO (git commit -s); open a PR whose description contains the line 'Fixes #16048' so the hive's contributor lane can auto-merge it once CI is green. Read and follow https://github.com/hivecommons/hive/blob/HEAD/AGENTS.md before changing code.
    

    Prefer not to drive the loop by hand? Run your own hive (we call them clankers) — a relay on your machine that claims, opens the PR and handles review fixes for you, and earns you a contributor profile and a leaderboard place: see https://hosted-kubestellar-hive-ojv6.hive.hivecommons.dev/contribute.

    If nobody claims it, hive may pick it up itself — claiming first prevents that. Docs: contributor relay · contributor lane auto-merge · trust tiers · AGENTS.md.

  14. added
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    stage/020-claimedClaimed by the scanner or a relay contributor
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Nov 2, 2026
  15. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

  16. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    Scanner lane 2026-11-02T14:50Z: no source change is needed for this incident. The red check came from the runner being lost partway through the Scan step. It did not come from gosec G-rule findings: Scan is continue-on-error: true, and every failed job has a null Scan conclusion and no log.

    The bounded-worker mitigation #16045 (gosec -concurrency 4) merged to v5 as 1e83f1f. The first v5 push run after it passed on attempt 1, with gosec finishing in about 2 min on self-hosted runner hive-runners-lke-qh55w-runner-rlnpd: https://github.com/hivecommons/hive/actions/runs/41460551983. The next v5 run on 2f96f44 (https://github.com/hivecommons/hive/actions/runs/41465174915) was still in progress at the time of this note.

    I'm not opening a duplicate PR. Following the maintainer decision, this stays open until recovery shows up across more v5 runs and same-repository PR runs.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  17. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Nov 2, 2026
  18. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by agent scanner released (holding agent relaunched).

  19. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    Scanner lane 2026-11-02T17:15Z: recovery confirmed — closing the incident.

    Since the bounded-worker mitigation #16045 (1e83f1f) landed on v5, every gosec (insecure patterns) run has finished green, on self-hosted runners, in minutes:

    The remaining red gosec is on #16188 (head 7fecd5b, 34 commits behind v5, run from before the mitigation); it just needs a fresh head — asked dependabot to rebase so the check re-runs on current v5. PRs stamped <!-- hive-shared-ci-16048 --> return to the normal queue now that the baseline is green.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerCreated or modified by the scanner agentapproved-directionciApproved by a Hive merger/owner for auto-merge on green CIgood first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/claimed-by-scanner

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions