Skip to content

perf: add fresh coverage scope pilot for mutation testing - #3506

Merged
thymikee merged 6 commits into
fix/mutation-sweep-completenessfrom
perf/mutation-coverage-scope-pilot
Oct 12, 2026
Merged

thymikee merged 6 commits into
fix/mutation-sweep-completenessfrom
perf/mutation-coverage-scope-pilot

Conversation

@thymikee

@thymikee thymikee commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Depends on #3475. Add an opt-in local coverage-scope pilot: a fresh whole-module probe selects runtime-covering test files for subsequent fresh mutation runs. Match module, epoch, seed, repository inputs, toolchain and environment; reject invalid explicit indexes.

Add reproducible fast-check seeds and scope/producer-cost reporting. Keep weekly and PR defaults broad; defer cross-run indexes, incremental results and static audits. Register the actual Stryker/Vitest hooks as test-framework entrypoints. Scope: 15 mutation tooling/configuration files.

Run pnpm mutation:run --help for the protocol and rollout criteria.

Validation

  • pnpm check:affected --run passed at 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb; mutation harness: 64/64 tests, including Node 22.12 with inherited environment state. Real Stryker verifies reporter loading without TypeScript flags; rebased fixtures use test-kit.
  • Missing provenance check planted: 62 pass / 1 fail; restored: 63 pass.
  • Seeded macOS canary measurement before final refactoring: all 390 identities preserved, no lost kills or additional uncovered mutants. One hit-limit timeout became an assertion kill; other 389 outcomes matched.
  • Consumer: 363s broad → 262s scoped (28% faster). Fresh producer: 145s; first producer-plus-consumer is 407s (12% slower). Costs must amortize across consumers.
  • Current CI passed all 17 selected shards and aggregation: 4,995 unique identities, zero harness errors. Five error shards preserve 391 identities; slowest 16m57s versus a prior deadline failure. All CI passed; no open review threads. Full same-head Ubuntu parity and net runner-minute/critical-path measurements are required before CI defaults change.

@thymikee
thymikee added this pull request to stack #3507 October 12, 2026 14:25
@thymikee thymikee changed the title perf/mutation coverage scope pilot perf: add fresh coverage scope pilot for mutation testing Oct 12, 2026
@thymikee

thymikee commented Oct 12, 2026 •

Copy link
Copy Markdown
Member Author

Consulted Claude on the public optimization proposal from #3475. The first follow-up adopts the fresh, opt-in producer and cost/provenance reporting; cross-run indexes, incremental result reuse and static audits remain separate work. Static audit gains in the earlier experiment came from changing execution policy with existing tests, not adding tests.

The implemented pilot is deliberately local: exact environments must match, and consumers run sequentially because scope/report paths are shared. Invalid explicit indexes fail visibly. A full fresh Ubuntu comparison is required before changing weekly defaults.

Seeded macOS canary experiment (seed 12345, before final refactoring):

Measurement Broad Scoped
Candidate/selected test files 358/358 358/122
Initial tests 2723 1100
Total consumer duration 362.995s 261.619s
Fresh mutant identities 390 390
Assertion kills 294 295
Hit-limit timeouts 5 4
Survived 83 83
Uncovered 8 8

The fresh producer cost 144.669s. Consumer-only saving is 27.9%; the first producer-plus-consumer costs 406.288s, 11.9% more than broad. This is not a net first-run speedup.

All 390 identities match, zero duplicate identities, no lost kills, no increased uncovered mutants. The other 389 statuses match. The single difference is the false ConditionalExpression at snapshot-occlusion.ts:439–441: broad hits the execution limit (201001/201000) before an assertion; scoped reaches the existing renderable-ancestor assertion and kills it. The same four named tests cover it, but their execution order changes.

Final head 612bbcf94211dcb51e4e60d175a669153f1ed173: all runnable affected gates passed, including 63 mutation harness tests. A planted missing identity check gave 62 pass / 1 fail, restored 63 pass. Final cleanup splits scope preparation/validation, loads fast-check only when a seed is supplied, declares actual test-framework hooks and keeps protocol help at its executable owner. No thresholds, exclusions or baselines were weakened.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.17 MB 5.17 MB +38 B
Package (unpacked) 5.17 MB 5.17 MB +38 B
Package (download) 1.55 MB 1.55 MB +14 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 23.1 ms 23.3 ms +0.3 ms
CLI --help 64.6 ms 64.6 ms +0.0 ms

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-3506/

Built to branch gh-pages at 2026-10-12 16:57 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 14 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/mutation/envelope.test.ts Resolved
Comment thread scripts/mutation/seed.test.ts Resolved
Comment thread docs/agents/testing.md Resolved
Comment thread scripts/mutation/coverage-scope.test.ts Resolved
Comment thread scripts/mutation/run.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

All five review findings are fixed, replied to and resolved at 2497b99199fd7183df8cdc3da77b87eec6323afd.

  • Node 22 subprocesses explicitly enable TypeScript stripping; the real Stryker plugin is plain JavaScript with JSDoc typing. A genuine loader smoke fails with the original TypeScript plugin and passes without child TypeScript flags after the migration.
  • Scoped pilot attempts emit coverage-scoped, including failed valid requests; probes emit coverage-probe.
  • The environment sentinel preserves inherited values, and the scope reference names KERNEL_MODULES.

All 64 harness tests passed on Node 22.12 with MUTATION_SCOPE_INPUT_TEST=changed. The environment collision and wrong envelope scope both reproduced before their fixes. The original seed worker test failed on Node 22.12 and now passes. Both stack heads passed pnpm check:affected --run; the worktree is clean. CI is rerunning on the updated heads.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the pilot. At 612bbcf the code looks sound, but I would like one change before this feeds any CI default. All 26 checks are green, and there are no conflicts. The mutation harness tests run under pnpm, so CI does not cover the Node 22.13-22.17 gap in the two open Cubic threads. Those threads need fixing before merge.

A scoped consumer must never report a mutant as NoCoverage that the probe saw as covered. Today coverage-scope.ts stores only the union of test files with any perTest hit, not the covered mutants. Stryker then labels every mutant without a covering test as NoCoverage. A test file with zero probe hits can still have hits in the consumer when its coverage depends on randomness the seed does not fix, such as explicit fc seeds, Math.random or time. That file is dropped, and its mutants show as uncovered instead of killed. Could you record the covered mutant ids from mutantCoverage.perTest and .static in CoverageScope? After the consumer report is read, fail the run if any NoCoverage mutant id is in that set.

Not blocking, and you can take or leave it: I did not confirm whether the Stryker 10 vitest-runner records zero perTest hits for a test that only runs module code at import time, so I do not know how wide that window is.

I did not rerun the harness tests or the planted-provenance mutation, because this review was read-only. The macOS canary parity numbers (390 identities, 363s to 262s) are author-reported, and I saw no artifact for them.

On the open threads: the envelope test and the seed test both hold. They run node without --experimental-strip-types, so please use process.execPath plus the flag. The window is Node 22.13-22.17. The env var thread and the scope field thread still apply at low priority. The anchor thread does not apply: docs/agents/testing.md already names scripts/mutation/modules.ts, and that line only changed by rewrapping, so please resolve it.

Next, please fix the two strip-types threads. Add the probe-versus-consumer NoCoverage check before any CI default changes.

A newer head, 2497b99, arrived after this review. Its delta looks like it addresses the strip-types and scope field threads, but it does not change coverage-scope.ts, so the NoCoverage question above still applies. I will check the new head separately.

@thymikee
thymikee force-pushed the perf/mutation-coverage-scope-pilot branch from 2497b99 to 086d0d2 Compare October 12, 2026 15:58
@thymikee

Copy link
Copy Markdown
Member Author

Rebased onto the validated #3475 head and migrated both pilot fixture imports to @agent-device/test-kit/tmp-dir.

Validated head 086d0d230e2cc6a465c52ca5e160de1654a15128: build, repository format, all 64 harness tests on Node 22.12 with inherited environment state, and pnpm check:affected --run passed. The real Stryker reporter-loading test also passes without TypeScript flags. Fresh CI has passed harness selection/self-tests and is running mutants. All review threads are resolved.

@thymikee
thymikee force-pushed the perf/mutation-coverage-scope-pilot branch from 086d0d2 to 617b5f4 Compare October 12, 2026 16:56
@thymikee

Copy link
Copy Markdown
Member Author

The fresh Ubuntu run exposed a second deadline cause: kernel-errors shard 2/3 spent 8m15s on its 10,246-test dry run, then reached only 80% mutation progress before the 22-minute step limit. Aggregation correctly failed with 13/14 reports and retained the partial table. Failing job.

The parent now uses five error shards, aligned with the weekly matrix and its 18-report completeness count. Native Stryker instrumentation preserves all 391 error mutant identities exactly once, with partitions 76/75/72/75/93 instead of 129/128/134. No test exclusions, source removals, score-policy changes or timeout extensions. This adds two dry runs in exchange for shorter mutation work per job and deadline margin.

Both exact heads pass pnpm check:affected --run: parent ff85105bab046bab3df6f61bef076d987263a9b8, pilot 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb. All 54 parent harness tests pass; all 64 pilot tests also pass on Node 22.12. Fresh CI is running.

@thymikee

Copy link
Copy Markdown
Member Author

The PR is ready from the code side at 617b5f4. CI is still running, so the Mutants shards need to finish green, since this diff touches the mutation Vitest setup and runStryker. After that, the base PR #3475 needs to merge. No conflicts.

Not blocking, and you can take or leave these: run.ts grows from 546 to 761 lines because the pilot argument checks, identity construction and probe finish live there, and one entry in coverage-scope.ts (for example resolvePilotTestScope) would keep that code with the concept it owns. delete process.env.npm_lifecycle_script in run.ts runs on every Stryker call, not only pilot runs, so it could apply only when a pilot flag is set, or the PR body could say so. scopeFingerprint hashes only tracked files and six pinned package versions, so gitignored inputs such as built workspace output can change without changing it. --help could say that, or the fingerprint could hash the lockfile-resolved store path.

I did not run Stryker or the harness tests. The parity numbers (390 identities, 28% consumer speedup) and the Node 22.12 results are your reports, and I did not check them. I also did not confirm that --dryRunOnly calls onDryRunCompleted in Stryker 10.0.0.

The open thread on the Node 22.12 seed.test.ts run still stands (#3506 (comment)). Four other threads no longer apply, so please resolve them: the envelope.test.ts Node 22.12 thread, because the PR body reports those runs passing at head (#3506 (comment)); the testing.md link thread, because it now links modules.ts#KERNEL_MODULES (#3506 (comment)); the coverage-scope.test.ts env var thread, because the test uses 'changed-again' and restores the prior value (#3506 (comment)); and the envelope data.scope thread, because run.ts writes pilotScope and envelope.test.ts:44 asserts it (#3506 (comment)).

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 12, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Fresh Ubuntu validation confirms the capacity fix at parent ff85105bab046bab3df6f61bef076d987263a9b8 and pilot 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb: all five error shards passed. Their native reports contain all 391 unique error mutant identities exactly once, and the reported source matches the current head. Shard sizes: 76/75/72/75/93; mutation steps: 11m06s, 9m19s, 16m57s, 13m46s and 15m24s. Slowest leaves 5m03s below the unchanged 22-minute mutation limit.

The current pilot run passed all 17 selected shards and aggregation: six modules, 4,995 unique raw identities, identical tested-merge provenance across reports and zero harness errors. These remain advisory mutation outcomes: 3,628 assertion kills, 1,122 survivors, 193 uncovered, 33 timeouts and 19 ignored. Coverage, Integration, all other main CI and all pilot platform checks passed.

All review threads on both PRs are resolved. The parent’s final iOS smoke job is still running; its main CI and mutation canary/aggregation have passed.

@thymikee
thymikee merged commit 396a235 into main Oct 12, 2026
42 checks passed
@thymikee
thymikee deleted the perf/mutation-coverage-scope-pilot branch October 12, 2026 17:46
@thymikee

Copy link
Copy Markdown
Member Author

All CI is now green on both exact pushed heads: #3475 ff85105bab046bab3df6f61bef076d987263a9b8 and #3506 617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb. The parent’s final iOS Settings, gesture and fixture-backed E2E smoke completed successfully, including host-focus verification. Both have passing main CI, Coverage, Integration, Apple/Android/Linux smoke and complete mutation aggregation.

Both exact heads passed pnpm check:affected --run; parent 54/54 and pilot 64/64 harness tests also pass on Node 22.12. All review threads are resolved and both PRs are mergeable. The coverage-scope pilot remains opt-in; CI defaults still use broad fresh tests.

@thymikee

Copy link
Copy Markdown
Member Author

Final state refresh confirms both PRs were merged at 17:46 UTC. CI is green on both tested heads and every review thread is resolved. The earlier mergeability note described the open stack; the closed PRs now report stale/unknown mergeability. Main contains the five error shards and matching eighteen-report weekly completeness requirement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant