Repository navigation
Conversation
|
Consulted Claude on the public optimization proposal from #3475. The first follow-up adopts the fresh, opt-in producer and cost/provenance reporting; cross-run indexes, incremental result reuse and static audits remain separate work. Static audit gains in the earlier experiment came from changing execution policy with existing tests, not adding tests. The implemented pilot is deliberately local: exact environments must match, and consumers run sequentially because scope/report paths are shared. Invalid explicit indexes fail visibly. A full fresh Ubuntu comparison is required before changing weekly defaults. Seeded macOS canary experiment (seed 12345, before final refactoring):
The fresh producer cost 144.669s. Consumer-only saving is 27.9%; the first producer-plus-consumer costs 406.288s, 11.9% more than broad. This is not a net first-run speedup. All 390 identities match, zero duplicate identities, no lost kills, no increased uncovered mutants. The other 389 statuses match. The single difference is the false ConditionalExpression at Final head |
Size Report
Startup median (7 runs, lower is better):
|
|
There was a problem hiding this comment.
All reported issues were addressed across 14 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
612bbcf to
2497b99
Compare
|
All five review findings are fixed, replied to and resolved at
All 64 harness tests passed on Node 22.12 with |
|
Thanks for the pilot. At 612bbcf the code looks sound, but I would like one change before this feeds any CI default. All 26 checks are green, and there are no conflicts. The mutation harness tests run under pnpm, so CI does not cover the Node 22.13-22.17 gap in the two open Cubic threads. Those threads need fixing before merge. A scoped consumer must never report a mutant as NoCoverage that the probe saw as covered. Today coverage-scope.ts stores only the union of test files with any perTest hit, not the covered mutants. Stryker then labels every mutant without a covering test as NoCoverage. A test file with zero probe hits can still have hits in the consumer when its coverage depends on randomness the seed does not fix, such as explicit fc seeds, Math.random or time. That file is dropped, and its mutants show as uncovered instead of killed. Could you record the covered mutant ids from mutantCoverage.perTest and .static in CoverageScope? After the consumer report is read, fail the run if any NoCoverage mutant id is in that set. Not blocking, and you can take or leave it: I did not confirm whether the Stryker 10 vitest-runner records zero perTest hits for a test that only runs module code at import time, so I do not know how wide that window is. I did not rerun the harness tests or the planted-provenance mutation, because this review was read-only. The macOS canary parity numbers (390 identities, 363s to 262s) are author-reported, and I saw no artifact for them. On the open threads: the envelope test and the seed test both hold. They run node without --experimental-strip-types, so please use process.execPath plus the flag. The window is Node 22.13-22.17. The env var thread and the scope field thread still apply at low priority. The anchor thread does not apply: docs/agents/testing.md already names scripts/mutation/modules.ts, and that line only changed by rewrapping, so please resolve it. Next, please fix the two strip-types threads. Add the probe-versus-consumer NoCoverage check before any CI default changes. A newer head, 2497b99, arrived after this review. Its delta looks like it addresses the strip-types and scope field threads, but it does not change coverage-scope.ts, so the NoCoverage question above still applies. I will check the new head separately. |
2497b99 to
086d0d2
Compare
|
Rebased onto the validated #3475 head and migrated both pilot fixture imports to Validated head |
086d0d2 to
617b5f4
Compare
|
The fresh Ubuntu run exposed a second deadline cause: The parent now uses five error shards, aligned with the weekly matrix and its 18-report completeness count. Native Stryker instrumentation preserves all 391 error mutant identities exactly once, with partitions 76/75/72/75/93 instead of 129/128/134. No test exclusions, source removals, score-policy changes or timeout extensions. This adds two dry runs in exchange for shorter mutation work per job and deadline margin. Both exact heads pass |
|
The PR is ready from the code side at 617b5f4. CI is still running, so the Mutants shards need to finish green, since this diff touches the mutation Vitest setup and runStryker. After that, the base PR #3475 needs to merge. No conflicts. Not blocking, and you can take or leave these: I did not run Stryker or the harness tests. The parity numbers (390 identities, 28% consumer speedup) and the Node 22.12 results are your reports, and I did not check them. I also did not confirm that The open thread on the Node 22.12 |
|
Fresh Ubuntu validation confirms the capacity fix at parent The current pilot run passed all 17 selected shards and aggregation: six modules, 4,995 unique raw identities, identical tested-merge provenance across reports and zero harness errors. These remain advisory mutation outcomes: 3,628 assertion kills, 1,122 survivors, 193 uncovered, 33 timeouts and 19 ignored. Coverage, Integration, all other main CI and all pilot platform checks passed. All review threads on both PRs are resolved. The parent’s final iOS smoke job is still running; its main CI and mutation canary/aggregation have passed. |
|
All CI is now green on both exact pushed heads: #3475 Both exact heads passed |
|
Final state refresh confirms both PRs were merged at 17:46 UTC. CI is green on both tested heads and every review thread is resolved. The earlier mergeability note described the open stack; the closed PRs now report stale/unknown mergeability. Main contains the five error shards and matching eighteen-report weekly completeness requirement. |
Summary
Depends on #3475. Add an opt-in local coverage-scope pilot: a fresh whole-module probe selects runtime-covering test files for subsequent fresh mutation runs. Match module, epoch, seed, repository inputs, toolchain and environment; reject invalid explicit indexes.
Add reproducible fast-check seeds and scope/producer-cost reporting. Keep weekly and PR defaults broad; defer cross-run indexes, incremental results and static audits. Register the actual Stryker/Vitest hooks as test-framework entrypoints. Scope: 15 mutation tooling/configuration files.
Run
pnpm mutation:run --helpfor the protocol and rollout criteria.Validation
pnpm check:affected --runpassed at617b5f4737579ba0bc0d7e0f13cecdf9af1d09bb; mutation harness: 64/64 tests, including Node 22.12 with inherited environment state. Real Stryker verifies reporter loading without TypeScript flags; rebased fixtures use test-kit.