Repository navigation
🌱 ci: mutation report keeps its own time budget and says when it is partial - #17604
Conversation
…artial A 20-minute runner kill of the advisory gremlins step wrote no comment.md, so XXL PRs kept the <!-- hive-mutation --> comment from an earlier head (#17596) or got none (#17572), and the text could not tell a clean run from a timeout. The run phase now bounds each package by the remaining budget, always writes the report, names the head SHA, and lists the packages it did not finish. test-mutation-report.sh pins the three shapes with a stubbed gremlins; TRUSTED_SCRIPTS_TREE re-pinned. Refs #17602 Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
There was a problem hiding this comment.
Style/maintainability review at head 34c1ccf. Ran bash src/scripts/test-mutation-report.sh locally: all assertions pass; TRUSTED_SCRIPTS_TREE matches git rev-parse HEAD:src/scripts (e4047bc…). Temp-dir layout and pass/bad helpers follow the sibling test-check-*.sh scripts.
Wrong issue reference in the test rationale — src/scripts/test-mutation-report.sh:13 says "Before #17601's sibling PRs were triaged, the job timeout killed the step first…". #17601 is an unrelated agent-token bug; the mutation-timeout issue this PR fixes (and the one the changelog fragment and PR body cite) is #17602. Please change it to #17602 so the comment that explains why the budget assertion exists points at the right history.
Redundant counter — src/scripts/mutation-report.sh:112-117 introduces total alongside the existing i; both are incremented at the same point after the same [ -n "$pkg" ] || continue guard, so they are always equal. Reuse i in the budget line at :181 (or rename i to total) rather than carrying two names for one value.
Out of scope here (already noted in the PR body / #17602): .github/workflows/mutation-advisory.yml neither runs the new test nor lists src/scripts/mutation-report.sh in paths:.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review follow-ups (below
|
…l counter Fix review findings on #17604: the test header rationale now points at #17602 (the mutation-advisory timeout issue) rather than the unrelated #17601, and the package loop reuses `i` instead of a duplicate `total` counter that was always equal to it. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
One blocking defect, otherwise the change reads correct to me.
TRUSTED_SCRIPTS_TREE is pinned to the wrong tree at this head
.github/scripts/integration-governance.cjs:12 pins e4047bc25a5e4ee3078f772edc5974770d0fb07b. That is the src/scripts tree of the first commit (34c1ccfee). The follow-up commit fba8d52ad edited both src/scripts/mutation-report.sh and src/scripts/test-mutation-report.sh again, so at the PR head:
git rev-parse fba8d52ad:src/scripts → 5d965ad41a5898673930799efca57d03fe31a52d
verify() at integration-governance.cjs:100-102 compares HEAD:src/scripts to the literal and throws src/scripts tree … is not the pinned TRUSTED_SCRIPTS_TREE … — the error text itself says the PR must re-pin to git rev-parse HEAD:src/scripts. The candidate-side governance checks (dco, changelog, contents, attribution in integration-candidate.yml) will all fail on this head, and if it were merged as-is every later v6 candidate would fail the same way until someone re-pins. Severity: high (blocks integration), trivial fix: set the literal to 5d965ad41a5898673930799efca57d03fe31a52d in the same commit as any further src/scripts change — a re-pin in a separate commit is stale the moment the next review-fix lands.
The PR body says the hash was "re-pinned to the new src/scripts tree"; that was true before the review-fix commit and is not now.
What I verified as correct
src/scripts/mutation-report.sh:17isset -uo pipefail(no-e), sorc=$?after the( … timeout … gremlins … )subshell at:125-132sees the real exit code and 124/137 is classified before the script continues..github/workflows/mutation-advisory.yml:27-29checks out withfetch-depth: 0on apull_requestevent, soHEAD^2atmutation-report.sh:66resolves to the PR head as the comment intends; the|| rev-parse HEADfallback covers the test harness's non-merge checkout.date +%sarithmetic andtimeout --kill-afterare GNU coreutils, present onubuntu-latest.- Default budget 1020 s + 10 s kill grace + gremlins install leaves margin under
timeout-minutes: 20. - The test's
hang 2case drives exactly the 124 path and assertscomment.mdexists, which is the regression that motivated #17602.
Not a finding, just noting for the record: test-mutation-report.sh is not run by any workflow (the repo convention is cli-pin-bump.yml:95-100 running its test-*.sh siblings). The body already says that wiring is deferred to #17602 because it needs .github/workflows write; fine.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from One blocking issue: the Stale tree pin (
|
…eport-budget Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
The runtime-tests contract compares the pin against the merge ref's src/scripts tree, so the pin must follow v6, not the branch tip alone. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Read the full script at the PR head and the callers (.github/workflows/mutation-advisory.yml, .github/scripts/integration-governance.cjs). Looks correct to me.
What I verified:
src/scripts/mutation-report.sh:17isset -uo pipefail(no-e), so dropping the|| echoguard and readingrc=$?after the subshell (lines 126–134) is safe; a non-zero gremlins exit still falls through to the report.iis incremented before the budgetcontinue(lines 113–119), so theN of ${i}count in the budget note counts every changed package, including the ones never started.- The survivor parser (lines 142–148) skips a missing
pkg-N.jsonand swallowsjqerrors on a partially written one, so a package killed mid-run cannot break the report. head_shaviaHEAD^2works because the workflow checks out withfetch-depth: 0(mutation-advisory.yml:28); the test's single-parent fallback path is also exercised.TRUSTED_SCRIPTS_TREE→5d965ad41a5898673930799efca57d03fe31a52dequalsgit rev-parse <head>:src/scripts; this is the re-pin the guard atintegration-governance.cjs:102-103demands and that #17527 / #17480 did the same way.- Ran
src/scripts/test-mutation-report.shagainst the PR head: 19 assertions pass in ~3 s, no network.
One note for whoever picks up the needs-human remainder from the body: every other src/scripts/test-*.sh runs as an explicit step in the script-tests job of .github/workflows/v2-ci.yml (around lines 400–460), not in the job that consumes the script, and v2-ci.yml is mirrored into integration-technical-v2-ci.yml by generate-integration-ci.py --check (v2-ci.yml:152). The step for this test belongs there; the paths: additions to mutation-advisory.yml are a separate change.
Until that step lands, nothing in CI executes the new test — the body says so, just making it visible here.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Test Improvement
Mutation (advisory)killsmutation-report.sh runattimeout-minutes: 20, and the script only wrotecomment.mdafter every changed package had finished. On size/XXL PRs that meant nothing was written: #17596 kept the<!-- hive-mutation -->comment from its first push (f9e3d3a) through two later heads, and #17572 never got one. 4 of the last 40 runs ended this way, all on the largest PRs — the place the gate was accepted (#17349) to look. The comment text also hedged(or the run hit its time budget…), so a clean run and a timeout read the same.src/scripts/mutation-report.sh: the run phase keepsMUTATION_BUDGET_SECONDS(default 1020 s, under the job's 20 min) and runs each package's gremlins undertimeoutwith what is left. A package killed by the budget is recorded as unfinished, later ones as not started, and the report is always written with the PR head SHA and, when truncated, a⏱ Time budget exhausted: N of M changed package(s)…list. Clean runs say so without the hedge.src/scripts/test-mutation-report.sh: stubbed gremlins + throwaway repo; pins survivors-in-changed-files (and filtering of untouched files), the clean message, budget exhaustion (comment still written, both packages listed, gremlins not started after the deadline), and the tests-only skip. Fails on the previous script (10 assertions), passes on this one; ~2 s, no network..github/scripts/integration-governance.cjs:TRUSTED_SCRIPTS_TREEre-pinned to the newsrc/scriptstree.Refs #17602 (needs-human: the workflow step that runs
test-mutation-report.shin CI, and the twopaths:entries so a change to the report script triggers the job, live in.github/workflows/mutation-advisory.yml, which this agent's contributor-tier token cannot push — the exact text is in the issue). Merging this PR ticks the first box of #17602; the workflow box remains.Filed by quality agent (ACMM L4/L6 — full mode)
— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88