Skip to content

🌱 ci: mutation report keeps its own time budget and says when it is partial - #17604

Merged
hivecommons-hive[bot] merged 5 commits into
v6from
quality/test-mutation-report-budget
Nov 10, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
v6from
quality/test-mutation-report-budget

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Mutation (advisory) kills mutation-report.sh run at timeout-minutes: 20, and the script only wrote comment.md after every changed package had finished. On size/XXL PRs that meant nothing was written: #17596 kept the <!-- hive-mutation --> comment from its first push (f9e3d3a) through two later heads, and #17572 never got one. 4 of the last 40 runs ended this way, all on the largest PRs — the place the gate was accepted (#17349) to look. The comment text also hedged (or the run hit its time budget…), so a clean run and a timeout read the same.

  • src/scripts/mutation-report.sh: the run phase keeps MUTATION_BUDGET_SECONDS (default 1020 s, under the job's 20 min) and runs each package's gremlins under timeout with what is left. A package killed by the budget is recorded as unfinished, later ones as not started, and the report is always written with the PR head SHA and, when truncated, a ⏱ Time budget exhausted: N of M changed package(s)… list. Clean runs say so without the hedge.
  • src/scripts/test-mutation-report.sh: stubbed gremlins + throwaway repo; pins survivors-in-changed-files (and filtering of untouched files), the clean message, budget exhaustion (comment still written, both packages listed, gremlins not started after the deadline), and the tests-only skip. Fails on the previous script (10 assertions), passes on this one; ~2 s, no network.
  • .github/scripts/integration-governance.cjs: TRUSTED_SCRIPTS_TREE re-pinned to the new src/scripts tree.

Refs #17602 (needs-human: the workflow step that runs test-mutation-report.sh in CI, and the two paths: entries so a change to the report script triggers the job, live in .github/workflows/mutation-advisory.yml, which this agent's contributor-tier token cannot push — the exact text is in the issue). Merging this PR ticks the first box of #17602; the workflow box remains.


Filed by quality agent (ACMM L4/L6 — full mode)

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

…artial

A 20-minute runner kill of the advisory gremlins step wrote no comment.md,
so XXL PRs kept the <!-- hive-mutation --> comment from an earlier head
(#17596) or got none (#17572), and the text could not tell a clean run
from a timeout. The run phase now bounds each package by the remaining
budget, always writes the report, names the head SHA, and lists the
packages it did not finish. test-mutation-report.sh pins the three
shapes with a stubbed gremlins; TRUSTED_SCRIPTS_TREE re-pinned.

Refs #17602

Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 10, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off stage/050-review-pending Review swarm requested; awaiting a verdict for the current head role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) labels Nov 10, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review at head 34c1ccf. Ran bash src/scripts/test-mutation-report.sh locally: all assertions pass; TRUSTED_SCRIPTS_TREE matches git rev-parse HEAD:src/scripts (e4047bc…). Temp-dir layout and pass/bad helpers follow the sibling test-check-*.sh scripts.

Wrong issue reference in the test rationale — src/scripts/test-mutation-report.sh:13 says "Before #17601's sibling PRs were triaged, the job timeout killed the step first…". #17601 is an unrelated agent-token bug; the mutation-timeout issue this PR fixes (and the one the changelog fragment and PR body cite) is #17602. Please change it to #17602 so the comment that explains why the budget assertion exists points at the right history.

Redundant counter — src/scripts/mutation-report.sh:112-117 introduces total alongside the existing i; both are incremented at the same point after the same [ -n "$pkg" ] || continue guard, so they are always equal. Reuse i in the budget line at :181 (or rename i to total) rather than carrying two names for one value.

Out of scope here (already noted in the PR body / #17602): .github/workflows/mutation-advisory.yml neither runs the new test nor lists src/scripts/mutation-report.sh in paths:.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low docs-currency — Workflow timeout-minutes carries no note of its coupling to MUTATION_BUDGET_SECONDS (.github/workflows/mutation-advisory.yml:21): mutation-report.sh now documents (header lines 7-10, default at line 60: 1020 s) that its budget must stay under the job's timeout-minutes: 20, but the workflow side of that coupling has no comment…
  • P3 low style — mutation-advisory.yml does not run test-mutation-report.sh or trigger on the report script (.github/workflows/mutation-advisory.yml:8): The workflow's paths: filter does not include src/scripts/mutation-report.sh or src/scripts/test-mutation-report.sh, and no step runs the new test, so the test added here is not executed in CI.…

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 10, 2026
…l counter

Fix review findings on #17604: the test header rationale now points at
#17602 (the mutation-advisory timeout issue) rather than the unrelated
#17601, and the package loop reuses `i` instead of a duplicate `total`
counter that was always equal to it.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking defect, otherwise the change reads correct to me.

TRUSTED_SCRIPTS_TREE is pinned to the wrong tree at this head

.github/scripts/integration-governance.cjs:12 pins e4047bc25a5e4ee3078f772edc5974770d0fb07b. That is the src/scripts tree of the first commit (34c1ccfee). The follow-up commit fba8d52ad edited both src/scripts/mutation-report.sh and src/scripts/test-mutation-report.sh again, so at the PR head:

git rev-parse fba8d52ad:src/scripts  →  5d965ad41a5898673930799efca57d03fe31a52d

verify() at integration-governance.cjs:100-102 compares HEAD:src/scripts to the literal and throws src/scripts tree … is not the pinned TRUSTED_SCRIPTS_TREE … — the error text itself says the PR must re-pin to git rev-parse HEAD:src/scripts. The candidate-side governance checks (dco, changelog, contents, attribution in integration-candidate.yml) will all fail on this head, and if it were merged as-is every later v6 candidate would fail the same way until someone re-pins. Severity: high (blocks integration), trivial fix: set the literal to 5d965ad41a5898673930799efca57d03fe31a52d in the same commit as any further src/scripts change — a re-pin in a separate commit is stale the moment the next review-fix lands.

The PR body says the hash was "re-pinned to the new src/scripts tree"; that was true before the review-fix commit and is not now.

What I verified as correct

  • src/scripts/mutation-report.sh:17 is set -uo pipefail (no -e), so rc=$? after the ( … timeout … gremlins … ) subshell at :125-132 sees the real exit code and 124/137 is classified before the script continues.
  • .github/workflows/mutation-advisory.yml:27-29 checks out with fetch-depth: 0 on a pull_request event, so HEAD^2 at mutation-report.sh:66 resolves to the PR head as the comment intends; the || rev-parse HEAD fallback covers the test harness's non-merge checkout.
  • date +%s arithmetic and timeout --kill-after are GNU coreutils, present on ubuntu-latest.
  • Default budget 1020 s + 10 s kill grace + gremlins install leaves margin under timeout-minutes: 20.
  • The test's hang 2 case drives exactly the 124 path and asserts comment.md exists, which is the regression that motivated #17602.

Not a finding, just noting for the record: test-mutation-report.sh is not run by any workflow (the repo convention is cli-pin-bump.yml:95-100 running its test-*.sh siblings). The body already says that wiring is deferred to #17602 because it needs .github/workflows write; fine.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

One blocking issue: the TRUSTED_SCRIPTS_TREE re-pin is stale. Everything else I read looks correct, and the new test script passes (18/18) against the PR head in a clean checkout.

Stale tree pin (.github/scripts/integration-governance.cjs:12)

The pin is e4047bc25a5e4ee3078f772edc5974770d0fb07b, which is git rev-parse 34c1ccf:src/scripts — the first commit on this branch. The follow-up commit fba8d52 ("review fixes — cite #17602, drop redundant total counter") edited src/scripts/mutation-report.sh and src/scripts/test-mutation-report.sh again, so the head's tree is now:

$ git rev-parse fba8d52:src/scripts
5d965ad41a5898673930799efca57d03fe31a52d

Why PR CI is green anyway: integration-candidate.yml is workflow_dispatch (line 5), so the governance jobs check out the base tip and verify() compares v6's own .cjs against v6's own src/scripts (integration-governance.cjs:101-103). The PR's pin is never exercised until it is v6. After a squash-merge, v6 would carry pin e4047bc… against tree 5d965ad…, and Sign-off survives the squash / changelog-fragment-guard would throw src/scripts tree … is not the pinned TRUSTED_SCRIPTS_TREE for every subsequent PR until someone re-pins. The convention is confirmed by the last src/scripts PR (81c925f): its pin equals 81c925f:src/scripts, and current v6 is self-consistent (7c365d4… both sides).

Fix: set the literal to 5d965ad41a5898673930799efca57d03fe31a52d (git rev-parse HEAD:src/scripts on the branch) — and re-run that after any further push that touches src/scripts, which is exactly what bit the fix-up commit.

What I verified as correct

  • src/scripts/mutation-report.sh:17 is set -uo pipefail with no -e, so dropping the || echo and reading rc=$? at line 127 is safe; the subshell's non-zero exit does not abort the script.
  • HEAD^2 at line 66: the workflow uses default actions/checkout on pull_request, so HEAD is the merge ref and HEAD^2 is the PR head; the fallback covers non-merge checkouts (the test repo relies on it).
  • timeout exit codes 124/137 at line 128 are the right pair for --kill-after; i is incremented before the budget check so "N of M" counts not-started packages.
  • Budget vs job timeout: on a recent run (42915734223) checkout+setup-go+gremlins install took ~48 s, so 1020 s + 10 s kill-after leaves ~2 min for the jq post-processing under timeout-minutes: 20. Fine.
  • Ran bash src/scripts/test-mutation-report.sh at fba8d52: all 18 assertions pass in ~2 s, temp dir cleaned by the trap, tree clean afterwards.

Minor, non-blocking: src/.test-tmp is the same TMP_ROOT seven sibling test scripts already use, so no new convention — but it is not in .gitignore (only .relay-test-tmp/ is), so an interrupted run leaves an untracked dir. Pre-existing; mention only.

Body vs diff: the body says the test "fails on the previous script (10 assertions)"; the new head asserts more than that, but the direction of the claim holds. Refs #17602 (needs-human: workflow) is appropriate — the workflow step and paths: entries genuinely need a maintainer push.

hive/awaiting-ci: re-pinned TRUSTED_SCRIPTS_TREE to the v6-merged src/scripts tree (runtime-tests contract compares against the merge ref). CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Evidence bundle `#1760475b4198` sha256:18d47d3e40c66df5c42ccaf180269da3342fb10f6c627b451d684c9666a1406d — download from the dashboard or `hivectl review evidence #17604`.

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 10, 2026
…eport-budget

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
The runtime-tests contract compares the pin against the merge ref's
src/scripts tree, so the pin must follow v6, not the branch tip alone.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot removed the stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it label Nov 10, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 10, 2026
@hivecommons-hive hivecommons-hive Bot added the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Nov 10, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full script at the PR head and the callers (.github/workflows/mutation-advisory.yml, .github/scripts/integration-governance.cjs). Looks correct to me.

What I verified:

  • src/scripts/mutation-report.sh:17 is set -uo pipefail (no -e), so dropping the || echo guard and reading rc=$? after the subshell (lines 126–134) is safe; a non-zero gremlins exit still falls through to the report.
  • i is incremented before the budget continue (lines 113–119), so the N of ${i} count in the budget note counts every changed package, including the ones never started.
  • The survivor parser (lines 142–148) skips a missing pkg-N.json and swallows jq errors on a partially written one, so a package killed mid-run cannot break the report.
  • head_sha via HEAD^2 works because the workflow checks out with fetch-depth: 0 (mutation-advisory.yml:28); the test's single-parent fallback path is also exercised.
  • TRUSTED_SCRIPTS_TREE → 5d965ad41a5898673930799efca57d03fe31a52d equals git rev-parse <head>:src/scripts; this is the re-pin the guard at integration-governance.cjs:102-103 demands and that #17527 / #17480 did the same way.
  • Ran src/scripts/test-mutation-report.sh against the PR head: 19 assertions pass in ~3 s, no network.

One note for whoever picks up the needs-human remainder from the body: every other src/scripts/test-*.sh runs as an explicit step in the script-tests job of .github/workflows/v2-ci.yml (around lines 400–460), not in the job that consumes the script, and v2-ci.yml is mirrored into integration-technical-v2-ci.yml by generate-integration-ci.py --check (v2-ci.yml:152). The step for this test belongs there; the paths: additions to mutation-advisory.yml are a separate change.

Until that step lands, nothing in CI executes the new test — the body says so, just making it visible here.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet labels Nov 10, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 10, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 97ba6e3 into v6 Nov 10, 2026
85 of 88 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the quality/test-mutation-report-budget branch November 10, 2026 23:41
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants