Skip to content

ci(repo): dry-run the main mutation lane once and keep its evidence - #155

Merged
ryanleecode merged 1 commit into
diag-l6-ci-recordsfrom
diag-l7-main-lane
Oct 2, 2026
Merged

ryanleecode merged 1 commit into
diag-l6-ci-recordsfrom
diag-l7-main-lane

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Layer 7 of the agent-ready failure diagnostics stack (docs/plans/2026-10-01-2306-feat-agent-ready-failure-diagnostics-plan.md, unit U17; R9, R10, R13, R14, R27, R28).

What changes

  • A new preflight job runs one dry run per planned package. If it fails, the lane stops and the job shows the failure record; if it passes, it publishes dry-run coverage that every shard reuses through the shared reports/stryker-incremental-*.json incrementalSources glob.
  • CLI: with --dryRunOnly and incremental mode on, a run now writes its dry-run coverage into its incremental file and keeps any verdicts already there. dryRunOnly leaves the run-inputs digest, so a preflight and the shards that reuse it compute the same digest. Existing caches are refused once with runInputsChanged.
  • Traces: each preflight and shard job starts LGTM, exports its traces with export-traces.ts the way the e2e lane does, and uploads them for 30 days.
  • Retention (KTD20): shard parts (report, stream, failure.json, SARIF) are kept 30 days. The report job deletes them only when every shard succeeded.
  • SARIF (KTD23): CI runs add the sarif reporter. The report job folds every package's SARIF into one run with repository-relative paths and re-indexed rules, then uploads it under the single category mutation. Several runs of one tool in one category are refused by code scanning, and per-shard categories would leave stale alerts open whenever the plan reshards.
  • Survivor baseline (KTD17): when every package's report is complete, main runs stryker gate --update-baseline on the merged report and publishes mutation-survivor-baseline.

Evidence

  • Shard reuse (KTD18 execution note): on stryker-js-typescript-checker with the workspace CLI, a preflight followed by shards 1/2 and 2/2 logged "Reusing the persisted dry-run coverage" in both shards. The shards' 498 verdicts match a cold run exactly.
  • Integration tests: incremental-reuse.integration.test.ts gains two scenarios.
    • The preflight spawns exactly the dry run a cold run spawns, and the run that reuses it spawns only its mutants. With publishing disabled, this scenario fails.
    • A dry-run-only run into an existing incremental file keeps every verdict there.
  • Pipeline smoke: run → combine → merge-reports → sarif → gate --update-baseline → gate ran locally on the published 15.0.0 CLI. Gating against a baseline missing one survivor exits 1.
  • Gates: pnpm check:ci passes and actionlint is clean.

Residuals

  • The dogfood lane runs the published CLI (AGENTS.md START-6). Until a release carries the dry-run-only coverage write, the preflight reports "not published by this CLI" and every shard keeps its own dry run, so R28 holds only after release.
  • The stryker annotate step is not added. Merged report paths are <package-slug>/<file> and per-package paths are package-relative, so the annotations would land on the wrong files until annotate can take a repository root.
  • On this layer, check (macos-latest) (run 36962074334) failed before any repository code ran: nix could not fetch dprint-aarch64-apple-darwin.zip 0.54.0 from github.com releases (HTTP 500 on all four attempts). The same job passed on ci(repo): run the mutation lane on pull requests and gate new survivors #156, which contains this layer. The CI token cannot rerun jobs.

A preflight job dry-runs every planned package and publishes the coverage that shards reuse through incrementalSources. A dry-run-only run now writes that coverage to its incremental file, keeping any verdicts there, and the run-inputs digest no longer covers dryRunOnly. Each job exports its traces, a failed shard's parts are kept 30 days, the report job uploads one merged SARIF run and publishes main's survivor baseline

Verdict-Semantics: unchanged

@ryanleecode ryanleecode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by the merge conductor as part of the diagnostics stack. CI green on this head except where noted in the stack summary (macOS timeouts on #151 are flaky: the same code is green on macOS in #155 and #156; the #156 stryker-js mutation shard times out until main records its first full-cost timing, as this PR's description says). Merging into feat/agent-ready-failure-diagnostics with a merge commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants