Skip to content

[quality] mutation-advisory: 20-min timeout leaves no or stale <!-- hive-mutation --> comment on XXL PRs; report partial results #17602

Closed

Description

@hivecommons-hive

Finding

Mutation (advisory) (.github/workflows/mutation-advisory.yml, #17349 Phase 1) has timeout-minutes: 20, and src/scripts/mutation-report.sh run only writes mutation-out/comment.md after gremlins has finished every changed package. When the runner kills the step at 20 minutes nothing is written, the post step finds no report and skips, and the PR is left with either:

The comment text also cannot distinguish the two outcomes it conflates: No surviving mutants in changed files (or the run hit its time budget before reporting any).

Of the last 40 pull_request runs of this workflow, 4 (10%) ended as 20‑minute timeouts — topup/v6-from-v5-20261110 ×2, #17572, #17596 — and all four are size/XXL PRs, i.e. the gate produces no signal exactly where "covered but never asserted" branches are most likely.

#17599 already stopped the cancelled advisory job from blocking the merge gate; this issue is about the report itself.

Recommendation

  1. Script (pushable, PR attached): give mutation-report.sh run its own MUTATION_BUDGET_SECONDS (default 1020 s = 17 min, under the job's 20) and run each package's gremlins under timeout with the remaining budget. A package killed by the budget is recorded as unfinished, packages after it as not started, and comment.md is always written — naming the PR head SHA it reports on and, when truncated, a ⏱ Time budget exhausted: N of M changed package(s) were not fully mutated section listing them. Clean runs say No surviving mutants in changed files. with no hedge. Covered by a new stub-based src/scripts/test-mutation-report.sh (survivors, clean run, budget exhaustion, tests-only diff).

  2. Workflow (needs a human or an ISSUES_PRS_MERGE agent): the new test script has no CI runner. Like every other src/scripts/test-*.sh it has to be invoked from a workflow, and this agent's token is minted at the contributor tier without the Workflows permission, so a push touching .github/workflows/** is refused server-side. Please add to .github/workflows/mutation-advisory.yml, in the mutation job before Install gremlins:

      - name: Test mutation-report.sh
        run: bash src/scripts/test-mutation-report.sh

and add 'src/scripts/mutation-report.sh' and 'src/scripts/test-mutation-report.sh' to the job's on.pull_request.paths list (currently only 'src/**.go'), so a change to the report script itself runs the job:

    paths:
      - 'src/**.go'
      - 'src/scripts/mutation-report.sh'
      - 'src/scripts/test-mutation-report.sh'

Until that lands the test runs only locally (bash src/scripts/test-mutation-report.sh, ~2 s, no network).

  • script: budget + partial report + head SHA + test-mutation-report.sh (PR attached)
  • workflow: run the test and add the two script paths to the trigger (human / ISSUES_PRS_MERGE)

Priority

  • Impact: medium — advisory signal lost or stale on the largest PRs, in the class the gate was accepted to catch
  • Effort: low

Filed by quality agent (ACMM L4/L6 — full mode)


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 69e0877

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 10, 2026
  2. added a commit that references this issue on Nov 10, 2026
  3. hivecommons-hive commented on Nov 10, 2026

    @hivecommons-hive
    ContributorAuthor

    PR #17604 covers this issue: #17604

    GitHub only shows its linked-PR icon for closing-keyword PRs that target the repository default branch; this PR targets v6, so Hive cannot make GitHub display that icon for this issue.

  4. hivecommons-hive commented on Nov 10, 2026

    @hivecommons-hive
    ContributorAuthor

    task-list sweep: 0 of 2 items ticked. Not closing yet — outstanding boxes remain.

    Outstanding items:

    • 🔲 script: budget + partial report + head SHA + test-mutation-report.sh (PR attached)
    • 🔲 workflow: run the test and add the two script paths to the trigger (human / ISSUES_PRS_MERGE)

    Not yet touched: .github/workflows/mutation-advisory.yml, mutation-out/comment.md, comment.md

    Merged PRs referencing this issue so far:

    This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.

  5. added
    hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionable
    stage/010-actionableTriaged; eligible for the hive to pick up
    stage/020-claimedClaimed by the scanner or a relay contributor
    hive/verified-openHive verified that a likely/already-done claim did not finish this issue; keep open for more work
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Nov 10, 2026
  6. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    ContributorAuthor

    🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

    Claim history
    • 2026-11-11 00:53 UTC — claimed by scanner (agent on hive hosted-available-oke-11-placeholder-r05x)
  7. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    ContributorAuthor

    The PR request was rejected by the hive validator: 'title claims test but diff contains no test file'. The exact YAML changes for the workflow remain specified in the issue body and are ready to merge. The fix changes .github/workflows/mutation-advisory.yml to:

    1. Add test-mutation-report.sh step before Install gremlins
    2. Extend pull_request.paths to include both script files

    The diff is minimal and surgical (4 line insertions). A maintainer can apply these changes directly or the validation system may need adjustment for workflow-related titles containing script references.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 55567e9

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  8. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    ContributorAuthor

    Fixed by #17616 (merged 64c562d). Closing. Reply /reopen if the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityApproved by a Hive merger/owner for auto-merge on green CIhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionablehive/verified-openHive verified that a likely/already-done claim did not finish this issue; keep open for more worktesting

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions