Skip to content

fix(mutation): complete weekly sweeps and validate harness changes - #3475

Merged
thymikee merged 15 commits into
mainfrom
fix/mutation-sweep-completeness
Oct 12, 2026
Merged

thymikee merged 15 commits into
mainfrom
fix/mutation-sweep-completeness

Conversation

@thymikee

@thymikee thymikee commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Weekly run 38114094784 exceeded deadlines and aggregated only 9/10 reports. Related discovery executed tests twice; partitions concentrated expensive mutation work.

  • Discover related tests without execution; retain per-test coverage. Balance actual mutants using the pinned Stryker instrumenter and complete statement boundaries: five error/eight selector shards. Exclude test fixture builders, preserving production mutants.
  • Use one completeness verdict for counts, module coverage, envelopes, and missing reports. Publish incomplete tables with uncovered/error/null fields.
  • Bound setup/mutation within 30-minute jobs; guard evidence/upload headroom and unconditional upload.
  • Rename the PR lane “Mutation Harness Canary”; include configuration/dependency/manifests. Scores remain advisory. Contributor guidance links authoritative scope declarations.

21 files changed. Five error shards leave deadline margin after the observed 8-minute dry run.

Validation

  • Head ff85105bab046bab3df6f61bef076d987263a9b8: pnpm check:affected --run passed every runnable gate; all 54 harness tests also passed on Node 22.12. Rebased onto d97c9bfbf3; use canonical test-kit fixtures after main’s helper migration.
  • Native instrumentation preserves 391 error/3,471 selector identities exactly. Largest partitions: errors 260→93; selectors 776→439. Unicode fixture preserves 12/12; real partial-range smoke passed 81 tests.
  • Full Ubuntu sweep passed all 16 shards and aggregation at 61a5b346d2 (historical evidence before the test-kit rebase). Verified 5,461 unique raw identities and matching source. Slowest step: 20m43s. Current canary passed through aggregation: 390 unique identities, zero errors, five timeouts; 13m53s. All current CI passed, including Coverage/Integration and Apple smoke. Five error shards passed: slowest 16m57s. All review threads resolved.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.17 MB 5.17 MB +47 B
Package (unpacked) 5.17 MB 5.17 MB +47 B
Package (download) 1.55 MB 1.55 MB -3 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 18.5 ms 22.0 ms +3.6 ms
CLI --help 54.0 ms 60.3 ms +6.3 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 19 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/mutation/run.ts Resolved
Comment thread scripts/mutation/score.ts Resolved
Comment thread scripts/mutation/workflow.test.ts Resolved
Comment thread .github/workflows/mutation-weekly.yml Resolved
Comment thread scripts/mutation/test-scope.test.ts Resolved
Comment thread scripts/mutation/config.test.ts Resolved
Comment thread scripts/mutation/workflow.test.ts Resolved
Comment thread docs/agents/testing.md Resolved
@thymikee

Copy link
Copy Markdown
Member Author

At dca5511, three open review threads still hold and need a fix before this is ready. The rest of the weekly sweep change looks sound.

Shard completeness is decided in three places in sweep() (run.ts L414). Because of this, a --report-dir run without --expect-shards skips the failed-envelope check, and the table banner and the thrown error come from different code. The rule is simple: with --report-dir, every merged report has a 'pass' envelope, the count matches when one is expected, and every module has mutants.

Could assertCompleteSweep(state, shards, args) own that rule and return one reason string or undefined? Then sweep() would print incomplete: reason and throw when it is set, and the try/catch, the error as Error cast and the duplicate count expression would go away. I found no smaller existing primitive for the range sharding, so sharding.ts with oxc-parser looks reasonable to me. If you prefer to keep the checks split, what would have to change first?

Three open inline threads still apply and are worth fixing before merge: the failed-envelope gap at run.ts:418 (thread), the missing noCoverage and errors nulls in the absent-module envelope (thread), and the 30-minute job timeout against the 25-minute step (thread). The other five threads still apply too; they are test-shape and docs gaps.

No conflicts. CI is still running: Mutants (snapshot-occlusion), Smoke Tests and Coverage are pending, with no failure yet. The Mutants job runs this PR's changed harness, so it is the direct check for this diff.

I did not confirm that cancelled() steps run after a job-level timeout, the Ubuntu runtime of the 15-shard sweep, or that oxc offsets match Stryker's columns on non-ASCII sources. I also did not reproduce the 391 / 3,471 mutant counts.

Before merge, please fold the failed-envelope check into assertCompleteSweep, add the two nulls, and fix the timeout budget. Then wait for the Mutants job to pass.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/mutation/workflow.test.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

Fixed and pushed at e3924c1f3f7e08936599c8ff03d08c94696085ac.

  • One completeness function now returns the reason used by the summary, stderr, and envelope; failed/missing envelopes are rejected with or without --expect-shards.
  • Unmeasured kernels explicitly emit noCoverage: null and errors: null.
  • Both jobs retain the repository's hard 30-minute maximum: checkout ≤1 minute, setup ≤2, mutation ≤22, leaving five minutes of configured evidence/upload headroom. The guard parses these budgets and requires unconditional upload. Planted budget/setup/upload faults fail.
  • The remaining naming/doc findings are fixed. Re-review's ordering concern was tested: moving either evidence step before mutation already fails the guard; the thread includes the counterfactual result.

pnpm check:affected --run passed on this exact head, including all 50 harness tests. Updated-head Mutants canary, Coverage, Integration Tests, Repo Guards, and review checks passed. All review threads are addressed. Native Stryker instrumentation also preserved 12/12 Unicode-fixture mutants across two partitions.

The full Ubuntu weekly sweep is running to measure all 15 jobs; I'll record its result separately.

@thymikee

Copy link
Copy Markdown
Member Author

The code now looks right at e3924c1, but I can't call this ready yet because the 15-job weekly sweep has not finished, so its runtime is unconfirmed. The earlier findings from dca5511 are fixed. The harness now fails an incomplete sweep, and the new tests cover the timeout budget and the always-run report upload.

I did not run the harness tests. I judged that the regression test fails without the fix by reading the dca5511 sweep() code. I also did not rerun the author's check that moving an evidence step before mutation fails workflow.test.ts. I did not verify whether GitHub runs cancelled() steps after a job-level timeout.

Smoke Tests is still in progress with no failure. The diff touches only the mutation scripts and workflows, stryker.config.json, vitest.mutation.config.ts and docs/agents/testing.md, which are not on the device or daemon route that Smoke Tests exercises. The author reports the Mutants canary passed on this head. There are no conflicts.

Not blocking: LANE_TOOLING is a hand-kept copy of the mutation-affected.yml pull_request paths filter. It grows from 3 to 10 entries here and is matched with node path.matchesGlob. The repo already has loadLanes, triggersOnPath and matchesGlob in scripts/gate/workflows.ts. Could affectedMatrixFor ask that module whether the diff triggers the mutation-affected lane? Then the workflow filter is the only list and the drift test in workflow.test.ts can go. You can take or leave this.

The other threads are all fixed at this head, so please resolve them: shards.failed rejection, absent-module branch, job timeout assertion, step timeout caps, test-scope rename, config test rename, upload if: always() and docs pointer to LANE_TOOLING. The step-order thread does not apply: a step moved before "Run mutants for" lands in the setup slice, which needs an integer timeout-minutes, so the guard still fails.

Before merge, the pending Smoke Tests jobs need to finish. The 15-job weekly sweep also needs to finish inside its 30-minute budget.

@thymikee

Copy link
Copy Markdown
Member Author

The full Ubuntu budget trial caught selector shard 8 and error shard 2 overrunning the mutant limit. Both failure envelopes were uploaded, and the report correctly failed as incomplete.

Fixed and published at 3b1a22f1037f93e8dc949ebd02b589d583dc02f8: syntax density now drives complete statement-boundary partitions. Native Stryker confirms the same 3,471 selector and 391 error mutants, without duplicates; the largest partitions dropped from 776 to 542 and from 260 to 172. Three error shards add setup cost but provide more budget margin.

The real timeout artifacts also revealed that envelope-only shards must be read independently of reports. The actual CLI regression failed before the fix and now rejects either a failed envelope or a passing envelope without its report, even without --expect-shards.

All runnable gates passed on this exact head, including 52 harness tests. The revised full Ubuntu sweep is running; final-head CI will provide the remaining timing evidence.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/mutation/modules.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

The weekly sweep at 3b1a22f still does not complete, so this needs one more change before merge. Run 38238796571 at this head failed on two shards. Each shard spends about 8m20s on the Stryker dry run before it mutates anything. After that, kernel-errors-2 (167 of 172 mutants) and selectors-2 (467 of 541) both hit the 22-minute step deadline.

The shard split balances syntax nodes, not wall time, so the shards are uneven. The kernel-errors shards hold 116, 172 and 103 mutants. Selectors-2 needed more than 22 minutes while its siblings finished in 7 to 11. Every weekly run at this head would publish an incomplete-sweep failure and no score. That misses the goal of this PR. The rule to satisfy is that for every matrix entry from shardMatrix(), the dry run plus that shard's mutant work fits the 22-minute step with margin. The shards field in modules.ts together with sharding.ts enumerates every entry. The smallest fix is to raise shards for selectors and kernel-errors. Balancing by native mutant count would also help, since it brings kernel-errors to about 131 per shard. Cutting the per-shard dry-run cost would work too. Then please rerun the full weekly workflow at the final head. That run must show every Mutants job green, including kernel-errors-* and selectors-*, and "Score the merged sweep" passing with --expect-shards equal to the matrix size.

The remaining open thread, on the third kernel-errors shard, does not apply anymore, so you can resolve it: #3475 (comment). The third shard is justified. At 3b1a22f even the 172-mutant shard of that 3-way split times out, so going back to 2 shards would be worse.

The Smoke Tests run still in progress is likely unrelated. This diff touches only the mutation scripts, workflows and config, and the docs. The failing check that matters here is the weekly run above. I did not reproduce the Stryker mutant counts or run the harness tests locally. I checked the changes by reading the code.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/mutation/sharding.ts Resolved
Comment thread scripts/mutation/sharding.ts Resolved
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-12 17:47 UTC

@thymikee

Copy link
Copy Markdown
Member Author

The earlier findings at 3b1a22f are now fixed at 4db7adc: the native-count planner replaces the syntax-density one, and all 16 Mutants shards in weekly run 38244490637 finished green, the largest (kernel-errors-2) at about 21m20s for the whole job. I found no new problems in the code. The one thing still missing is proof that the merged sweep scores correctly: the "Mutation score report" job in that run (or a rerun at 4db7adc) must finish green, with its "Score the merged sweep" step passing --expect-shards equal to the 16-entry matrix. That job is still queued, and the only change since the green shards at 61a5b34 is docs.

CI is pending. Smoke Tests are queued and in progress, and this diff does not touch what they exercise. Mutants (snapshot-occlusion) is queued and does run the changed harness, but it has not failed, and the same shard passed in the weekly run at the code-identical 61a5b34. I know of no conflicts. Once the report job passes, nothing else stops the merge.

On the open inline threads from cubic-dev-ai: the cost thread #3475 (comment) does not apply, because the 8m20s is the Stryker initial test run and the planner only instruments sources, which takes under a second. The shard-count thread #3475 (comment) does not apply, because the 20m43s kernel-errors-2 measurement shows a 2-way split would overrun, so shards: 3 in modules.ts is justified. The deadline thread #3475 (comment) is fixed at this commit and can be resolved.

I did not reproduce the per-shard mutant counts or the 391/3,471 totals locally. I did not run the harness unit tests. I read the identity guard at sharding.ts:97 but did not run it, and I did not check that Stryker locations are 0-based, which the offset() mapping in sharding.ts assumes. A wrong base would throw the span error rather than lose mutants silently, and the CI shards passed. The deadline margin is thin, since kernel-errors-2 reached about 20m43s against 22m. Runner variance could push a future run over, and one green run does not show headroom.

@thymikee

Copy link
Copy Markdown
Member Author

The full native-count Ubuntu sweep passed all 16 shards and aggregation: https://github.com/callstack/agent-device/actions/runs/38244490637. Its mutation code is identical to current head 4db7adc; only contributor guidance was shortened afterward. All 5,461 raw mutant identities are unique and every reported source matches the current checkout. Error shards took 15m38s/20m43s/9m35s; the heaviest selector shard took 19m58s. All 12 review threads are resolved. Every current-head PR check passed, including Coverage, Integration, and the complete canary (mutants: 15m05s): https://github.com/callstack/agent-device/actions/runs/38245625375.

There is measured room for a follow-up:

Local experiment Before After Signal retained or gained
Snapshot canary: coverage-derived test scope 4m57s 3m36s All 390 identities and outcomes match
Unchanged narrow incremental run 4m57s 49s 390 reused, zero freshly executed
Ref flag policy: coverage-derived scope plus static audit 88s 23s Same 24 identities; 13→23 kills, no lost baseline kills

These are temporary macOS experiments, not shipped changes or Ubuntu promises; an existing same-input broad report supplied the scope index. A one-file owner-only audit was rejected because it lost three runtime kills. Safe provenance/invalidation and a fresh broad reference remain necessary.

The passing Ubuntu sweep spent 74 of 176 mutation-step runner-minutes on initial suites (42%). Halving that component would recover about 37 runner-minutes before index overhead. Prioritize coverage-derived scope and focused static coverage, then conservative incremental PR feedback; retain fresh weekly sweeps. Reports should expose fresh/reused work, static/hybrid status, covering tests, and uncovered decisions. The current report has 197 uncovered mutants (166 in selectors), providing a concrete strengthening backlog before broad inventory expansion.

@thymikee

Copy link
Copy Markdown
Member Author

Holistic follow-up at 4db7adc9b8ddeb14ab9eefb1bb9edd2b2d74a32c: no new open review threads or actionable blocker found. All 12 threads are resolved; all 26 current PR checks passed or were intentionally skipped, and GitHub reports the PR conflict-free.

An independent read-only native-instrumenter audit compared unsharded versus partitioned output by file, location, mutator, replacement, and ignore reason: kernel-errors 391→391 across three shards, selectors 3,471→3,471 across eight, with zero missing, extra, or duplicated identities. Stryker's public instrumenter creates a fresh collector per invocation, and its range API uses zero-based locations; the OXC/native boundary mapping is consistent. The pinned instrumenter avoids maintaining a second mutation-weight model.

Reviewed the real CLI failure paths, scope discovery, scoring, workflow selection and budgets, dependency/lockfile scope, and the test ownership. Nonzero Stryker exits, missing/failed envelopes, report-less artifacts, missing module coverage, and short shard sets fail; available scores remain visible. Scope discovery does not execute tests, and per-test coverage remains enabled. The full 16-shard sweep and its aggregate passed, as did the final-head end-to-end canary. No source changes were needed in this follow-up.

The remaining operational caveat is runtime margin: kernel-errors-2 took 20m43s against the 22-minute step deadline (77 seconds remaining). One passing sweep does not establish a long-term runtime guarantee. Coverage-derived scopes and measured cost packing remain the supported follow-up, rather than changing the production mutation universe or hiding failures. The new main commit since the passing CI changes recording overlays; it does not require rebasing this mutation-only diff.

@thymikee

Copy link
Copy Markdown
Member Author

The missing evidence at 4db7adc is now in place. The "Mutation score report" job in weekly run 38244490637 passed, and its "Score the merged sweep" step ran with --expect-shards 16: https://github.com/callstack/agent-device/actions/runs/38244490637/job/114810976274. That run was at 61a5b34, and the only change since then is docs/agents/testing.md, so it covers the current head. I found no code problems, CI is green, and I know of no conflicts. This is ready for human review.

The 77-second margin on kernel-errors-2 against the 22-minute step deadline is not blocking, but a slower runner could tip it over, so the coverage-derived scope follow-up is worth tracking.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 12, 2026
@thymikee
thymikee added this pull request to stack #3507 October 12, 2026 14:25
@thymikee

Copy link
Copy Markdown
Member Author

Both PRs were rechecked. #3475 has no unresolved review threads. Testing the follow-up on the supported minimum Node 22.12 also exposed a parent test that incorrectly demanded warning-free stderr. Fixed at 0839f608752318588aa949030960b9bcde2ae70d: it now requires exactly the expected mutation-harness error alongside legitimate Node warnings. All 54 parent harness tests pass on Node 22.12, and pnpm check:affected --run passed on this exact pushed head.

The production mutation code is unchanged, so the previous full 16-shard Ubuntu sweep remains applicable. Updated-head CI is running. #3506 was rebased onto this parent and its five findings were fixed, replied to and resolved; its 64 harness tests also pass on Node 22.12 with inherited environment state, including a real Stryker reporter loader smoke.

@thymikee
thymikee force-pushed the fix/mutation-sweep-completeness branch from 0839f60 to ba9cfec Compare October 12, 2026 15:58
@thymikee

Copy link
Copy Markdown
Member Author

Rebased onto main at d97c9bfbf3 and fixed the CI root cause: the harness tests imported the retired temporary-directory helper after main moved it into @agent-device/test-kit. All three parent imports now use the owning package export; no compatibility shim.

Validated head ba9cfec7b471bbaf42b6233c05414a232189f2cd: build, all 54 harness tests, and pnpm check:affected --run passed. Fresh CI is running; the previous failure was the import error, before mutants started.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. At ba9cfec the code looks clean, and the earlier review at 4db7adc was also clean. The new commit only touches scripts/mutation/*.test.ts (imports and one assertion), so nothing in the harness, CI job definitions or docs changed since then. I did not re-run the harness tests or check:affected locally, so the pass result is yours. The earlier threads show open=0 and resolved=12, the same as at 4db7adc, so I read it as no thread changing since. CI is still pending: both Smoke Tests jobs are queued or running. This PR changes the mutation harness, CI job definitions and docs, and none of the daemon or device routes that smoke covers, so a smoke failure would most likely not come from it. No conflicts. Once Smoke Tests finish green on ba9cfec, the PR is ready for a human merge.

@thymikee

Copy link
Copy Markdown
Member Author

The fresh Ubuntu run exposed a second deadline cause: kernel-errors shard 2/3 spent 8m15s on its 10,246-test dry run, then reached only 80% mutation progress before the 22-minute step limit. Aggregation correctly failed with 13/14 reports and retained the partial table. Failing job.

The parent now uses five error shards, aligned with the weekly matrix and its 18-report completeness count. Native Stryker instrumentation preserves all 391 error mutant identities exactly once, with partitions 76/75/72/75/93 instead of 129/128/134. No test exclusions, source removals, score-policy changes or timeout extensions. This adds two dry runs in exchange for shorter mutation work per job and deadline margin.

Both exact heads pass pnpm check:affected --run: parent ff85105bab046bab3df6f61bef076d987263a9b8, pilot 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb. All 54 parent harness tests pass; all 64 pilot tests also pass on Node 22.12. Fresh CI is running.

@thymikee

Copy link
Copy Markdown
Member Author

The PR is ready from the code side. The earlier review at ba9cfec found nothing blocking, and this update at ff85105 still finds nothing blocking.

Not blocking: the weekly sweep now runs five shards of 76/75/72/75/93 mutants, and each shard still pays the roughly 8m15s dry run, so at about 8 s per mutant the 93-mutant shard 5/5 needs about 21 minutes against the 22-minute step limit. No CI run at this commit ran a kernel-errors shard, so that margin is an estimate. You can take it or leave it, but one dispatch of the weekly sweep (or forcing kernel-errors in the affected lane) would confirm that shard 5/5 finishes with room to spare.

I did not reproduce the 391-mutant identity preservation or the partition sizes, and I did not run the harness tests locally, so those are your results.

The diff touches only the mutation scripts, the GitHub Actions files and config, so it does not overlap the smoke route. Two Smoke Tests jobs are still pending, the other two passed, and all other checks pass, including the affected mutation lane. There are no conflicts. Once the two pending Smoke Tests jobs finish green, the PR is ready for human review.

@thymikee

Copy link
Copy Markdown
Member Author

Fresh Ubuntu validation confirms the capacity fix at parent ff85105bab046bab3df6f61bef076d987263a9b8 and pilot 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb: all five error shards passed. Their native reports contain all 391 unique error mutant identities exactly once, and the reported source matches the current head. Shard sizes: 76/75/72/75/93; mutation steps: 11m06s, 9m19s, 16m57s, 13m46s and 15m24s. Slowest leaves 5m03s below the unchanged 22-minute mutation limit.

The current pilot run passed all 17 selected shards and aggregation: six modules, 4,995 unique raw identities, identical tested-merge provenance across reports and zero harness errors. These remain advisory mutation outcomes: 3,628 assertion kills, 1,122 survivors, 193 uncovered, 33 timeouts and 19 ignored. Coverage, Integration, all other main CI and all pilot platform checks passed.

All review threads on both PRs are resolved. The parent’s final iOS smoke job is still running; its main CI and mutation canary/aggregation have passed.

@thymikee
thymikee merged commit 89c9a2d into main Oct 12, 2026
26 checks passed
@thymikee
thymikee deleted the fix/mutation-sweep-completeness branch October 12, 2026 17:46
@thymikee

Copy link
Copy Markdown
Member Author

All CI is now green on both exact pushed heads: #3475 ff85105bab046bab3df6f61bef076d987263a9b8 and #3506 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb. The parent’s final iOS Settings, gesture and fixture-backed E2E smoke completed successfully, including host-focus verification. Both have passing main CI, Coverage, Integration, Apple/Android/Linux smoke and complete mutation aggregation.

Both exact heads passed pnpm check:affected --run; parent 54/54 and pilot 64/64 harness tests also pass on Node 22.12. All review threads are resolved and both PRs are mergeable. The coverage-scope pilot remains opt-in; CI defaults still use broad fresh tests.

@thymikee

Copy link
Copy Markdown
Member Author

Final state refresh confirms both PRs were merged at 17:46 UTC. CI is green on both tested heads and every review thread is resolved. The earlier mergeability note described the open stack; the closed PRs now report stale/unknown mergeability. Main contains the five error shards and matching eighteen-report weekly completeness requirement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant