Skip to content

ci: distribute fresh mutations across verified native ranges - #53

Merged
choskeli merged 2 commits into
mainfrom
ci/balance-native-mutation-ranges
Oct 28, 2026
Merged

choskeli merged 2 commits into
mainfrom
ci/balance-native-mutation-ranges

Conversation

@choskeli

@choskeli choskeli commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Integration refresh

Reviewed balancing implementation is unchanged at refreshed head c6450a4511bad3e8a729a00a1bbc324a755e90dd (tree 7d5bd0ca338e1d9fd73aa7134de447eedea57982). This merges current main 853d878aa6fe242f151adfc0d1bc15083dd2529b once to satisfy strict freshness. The delta from accepted head 2a8db289 is exactly PR51's four files: .github/workflows/ci.yml, docs/testing.md, and the two quick-start smoke files. All 13 balancing-change files are byte-for-byte unchanged. No timing-weight implementation is included.

Required refreshed-head CI 40606063220 passed on c6450a4511bad3e8a729a00a1bbc324a755e90dd: 21 successful jobs and three intentional skips, with both required check contexts green. Parent and independent source reviews cleared this exact refresh. The final artifact SHA-256 is 5f78212048e5d6a349934c184158ae6f73626b0b15b9a1e483076965191c9bcf. Its audit independently reproduces: 1,289 exact native identities, all 331 declared input records, 31 test files, 1,224 killed / 34 timeout / 31 reviewed equivalents, 100% adjusted and zero reused results. The measured 8m32 acceptance below remains explicitly bound to its original commit; it is not relabelled as a run on this refreshed head.

Final balanced acceptance — passed

One full acceptance run, 40601378445, passed on frozen head 2a8db289c59e14041859294a1df34defe5fa4a70 (tree 8794e9c71427bc2f303eb83a9ad9b96be63d221a), attempt 1. All 34 jobs completed successfully or were intentionally skipped by the selected workflow path.

Measurement PR50 baseline Balanced acceptance
Complete CI run 13m24s 8m32s
Parser shard window (first shard start to last shard finish) 11m18s 6m32s
Full parser path (planning start through collector finish) 11m52s 7m16s
Median parser job 1m47s 2m21s
Sum of parser shard job durations (excludes planning/collection) 33m54s 29m40s
Sum of all parser job durations (includes planning/collection) 34m23s 30m19s
Planning job 11s 17s
Collector job 18s 22s

Observed total latency fell 36.3%; the parser window fell 42.2%, and summed parser shard job duration fell 12.5%. Total shard setup stayed 140s in both runs. Prerequisites were 19s faster, while planning and collection added 10s; the primary gain came from mutation execution. Summed native execution fell 1,466s → 1,197s (−18.3%); summed coverage dry runs rose 238.256s → 255.665s (+7.3%). Work remains uneven (max/median 2.78 versus 6.34), so this is a measured improvement, not an equal-runtime guarantee. The remaining largest cost is the complete 43-mutant byte-reader.ts:98–131 component; no follow-up source changes or additional acceptance runs were made.

Raw evidence independently re-audited: 1,289 mutants = 1,223 killed + 38 timeout + 28 reviewed equivalents; 100% adjusted, 0 reused. Every mutant identity and outcome matches the accepted baseline. All 329 declared input digests match frozen Git blobs; all 11 parser sources, 31 test files, report digests, run/attempt/commit provenance and source-bound proofs match. The hosted audit reproduces exactly from unchanged reports.

The separate ordinary PR CI run, 40600554437, passed on 2a8db289: all required checks succeeded. Its cold native report independently re-audits to 1,289 mutants, 1,226 killed / 32 timeout / 31 reviewed equivalents, 100% adjusted and zero reused. Six timeout outcomes differ from the balanced run; all pass the existing audit and source-bound proofs. Artifact digest, all 329 input records and all 31 test-file sources were checked. The ordinary execution path is unchanged.

PR50's fresh matrix still gives one entire hot source file to one runner. In accepted run 40591855465, free-format-probe.ts had only 183 mutants but took 10m25s inside Stryker: its 75s coverage dry run inflated three method timeouts and one constructor timeout to about 105s each. Whole-file regrouping cannot shorten that straggler.

This draft uses the already-locked native Stryker 10.1.1 instrumenter to discover complete overlapping mutant spans, then assigns larger components first to the least-populated of the same eleven parser jobs. Jobs use normal native --mutate ranges. Each assignment is re-instrumented, its complete union is checked, and collection compares every raw mutant identity with the freshly generated inventory. Missing, extra, duplicate or misplaced mutants fail. Raw reports, source-bound equivalence proofs, local test IDs, full test inventory, two workers and ordinary incremental/cache behavior are preserved.

The new planner is 107 physical lines; the inventory validator is 29 lines plus small wiring/count aggregation changes. The complete diff is 13 files, +424/-49, mostly fixtures and tests. The direct instrumenter dependency already existed transitively at exactly 10.1.1; no new package graph or service is introduced. Planning now installs the lockfile. No historical duration database, custom cache matching, dependency graph or report rewrite is added.

Native-tool evidence and limits

  • StrykerJS has no distributed mutant-ID/shard selector. Its pinned implementation requires complete AST-node containment; arbitrary line cuts can omit spanning mutants.
  • Maintained agent-device usage supports verified native ranges, but its measured shards also show why equal mutant counts do not mean equal runtime.
  • Current planning preserves all 1,289 mutants, produces 86 indivisible components, and packs 117–118 mutants per job. Full planning and native identity proof took 401ms locally.
  • Historical reported mutant duration sums range 8.6–476.7s across bins. These are advisory cost observations, not a wall-time prediction: native Killed/Survived durations sum executed test time, while timeout durations measure the timeout interval; setup, coverage dry runs and recovery add further cost. The expensive methods are separated, but a spanning method cannot be subdivided safely.

Short validation at 2a8db289c59e14041859294a1df34defe5fa4a70

Windows, Node 24.22.1 / npm 11.19.1:

  • Native single-file/two-range fixture passed in 13.4s: ten fresh mutants, complete audit, no cache promotion, rejection of missing mutants even after updating the report digest, changed inputs and failed execution.
  • Exact proposed shard 8 completed in 66.2s under a 90s cap: 117/117 identities, 108 killed / 7 timeout / 2 reviewed survivors, identical outcomes and source/test inventory to accepted raw reports. Its full 1,771-test coverage dry run took 11s. An earlier 11-mutant method diagnostic completed in 46.4s with identical outcomes. Hosted versus local timings are different machines and are not a controlled speedup claim.
  • 208 focused policy/workflow checks passed; independent review found no blocking defect and separately passed 49 planner/shard-policy checks. Formatting, lint, both TypeScript checks, build, lock verification and diff whitespace checks passed.
  • Broader coverage: 2,124 passed, six HTTP cleanup failures; all six reproduce on untouched merged main. Broader tooling: 526 passed, 114 failures, one skipped; every failing case reproduces on merged main (baseline has one additional failure). These are existing Windows/native-tool limitations, including POSIX executables and filesystem/socket behavior. Required Linux CI remains authoritative.

After review, exactly one final full acceptance run was dispatched on the frozen head; its successful result is recorded above. No source changes or repeated acceptance runs were needed. The refreshed commit passed its own required checks and final source review; no additional full matrix campaign was needed for the unchanged balancing implementation.

Coordination: no quickstart/docs-PR files are changed. The separate clarity task overlaps scripts/full-mutations.mjs and scripts/mutation-policy-inputs.mjs; this draft leaves helper names and existing scope diagnostic wording unchanged.

@choskeli
choskeli marked this pull request as ready for review October 28, 2026 00:32
@choskeli
choskeli merged commit d5d4790 into main Oct 28, 2026
31 checks passed
@choskeli
choskeli deleted the ci/balance-native-mutation-ranges branch October 28, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant