Skip to content

ci(repo): run the mutation lane on pull requests and gate new survivors - #156

Merged
ryanleecode merged 2 commits into
diag-l7-main-lanefrom
diag-l8-pr-lane
Oct 2, 2026
Merged

ryanleecode merged 2 commits into
diag-l7-main-lanefrom
diag-l8-pr-lane

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Layer 8 of the agent-ready failure diagnostics stack (docs/plans/2026-10-01-2306-feat-agent-ready-failure-diagnostics-plan.md, unit U18; R23-R26, AE2, AE4, AE6, F2, F3).

What changes

  • mutation.yml now also runs on pull_request, with the same plan, preflight and sharded matrix as main. A pull request restores main's verdict cache read-only: it saves no cache, records no timings and publishes no baseline. It therefore evaluates only the mutants whose reused verdicts its change invalidates (R24).
  • The report job downloads the latest mutation-survivor-baseline from main and runs stryker gate --baseline on the merged report. A survivor absent from the baseline fails the job with the gate's record (R25). No step passes --update-baseline on a pull request, so no automated actor accepts a survivor (R26).
  • When every verdict was reused, the combine step writes "evaluated no mutants" to the summary and raises a notice, so the run does not read as a pass (R24, AE6).

Deviation from the plan

KTD16 runs a pull request's mutants unsharded and shards only if a replay of the largest recent PR exceeds the 15-minute target. This layer reuses main's timing-based plan for every PR instead, which means:

  • The 15-minute target holds by construction.
  • The check names stay the ones main already uses.

Evidence

pnpm check:ci passes and actionlint is clean. This PR's own run of the Mutation workflow is the in-PR proof: preflight, shards, merge and gate. Until main publishes its first baseline, the gate step warns that it was skipped rather than failing.

In-PR run (Mutation run 36962080291)

  • Plan → preflight → shards → report: the preflight dry-ran all four packages (stryker-js: 1017 tests) and printed "not published by this CLI", as expected on the published 15.0.0. The combine step reported 7465 evaluated mutants. The gate warned that main has no baseline yet.
  • stryker-js shard timeout: the plan budgeted this shard at 9m43s, because the timing record held main's reuse-shrunk duration. This stack invalidates most stryker-js verdicts, so the shard hit the 1800 s cap. Its summary rendered the JobTimedOut record with no infrastructure guesswork.
  • Fix (ci(repo): plan mutation shards from the full mutant set's cost):
    • Each timing entry now carries the reuse line's ran and reused.
    • mergeRecord scales a run to the full mutant set, and keeps the previous value when nothing ran.
    • Shards skip the preflight download (and its ##[error] line) when no coverage was published.
  • Until main records a timing with the new fields, PR plans still read the old record, so this PR's own stryker-js shard can time out again.

A pull request runs the same preflight and sharded incremental run against main's restored verdict cache, saves no cache and records no timings, then gates its merged report against the survivor baseline main published. A run that evaluated no mutants says so in a notice instead of reading as a pass

Verdict-Semantics: unchanged
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

This stack's own pull request ran 7465 invalidated stryker-js mutants in one job planned at 9m43s and hit the 1800s cap: the timing record held main's reuse-shrunk duration. Each entry now carries the reuse line's ran and reused counts, the record scales a run to the full mutant set and keeps its previous value when nothing ran, and shards download the preflight coverage only when the preflight published some

Verdict-Semantics: unchanged

@ryanleecode ryanleecode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by the merge conductor as part of the diagnostics stack. CI green on this head except where noted in the stack summary (macOS timeouts on #151 are flaky: the same code is green on macOS in #155 and #156; the #156 stryker-js mutation shard times out until main records its first full-cost timing, as this PR's description says). Merging into feat/agent-ready-failure-diagnostics with a merge commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants