Repository navigation
fix(mutation): complete weekly sweeps and validate harness changes - #3475
Conversation
Size Report
Startup median (7 runs, lower is better):
|
There was a problem hiding this comment.
All reported issues were addressed across 19 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
At dca5511, three open review threads still hold and need a fix before this is ready. The rest of the weekly sweep change looks sound. Shard completeness is decided in three places in Could Three open inline threads still apply and are worth fixing before merge: the failed-envelope gap at run.ts:418 (thread), the missing No conflicts. CI is still running: Mutants (snapshot-occlusion), Smoke Tests and Coverage are pending, with no failure yet. The Mutants job runs this PR's changed harness, so it is the direct check for this diff. I did not confirm that Before merge, please fold the failed-envelope check into |
There was a problem hiding this comment.
All reported issues were addressed across 8 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
Fixed and pushed at
The full Ubuntu weekly sweep is running to measure all 15 jobs; I'll record its result separately. |
|
The code now looks right at e3924c1, but I can't call this ready yet because the 15-job weekly sweep has not finished, so its runtime is unconfirmed. The earlier findings from dca5511 are fixed. The harness now fails an incomplete sweep, and the new tests cover the timeout budget and the always-run report upload. I did not run the harness tests. I judged that the regression test fails without the fix by reading the dca5511 sweep() code. I also did not rerun the author's check that moving an evidence step before mutation fails workflow.test.ts. I did not verify whether GitHub runs cancelled() steps after a job-level timeout. Smoke Tests is still in progress with no failure. The diff touches only the mutation scripts and workflows, stryker.config.json, vitest.mutation.config.ts and docs/agents/testing.md, which are not on the device or daemon route that Smoke Tests exercises. The author reports the Mutants canary passed on this head. There are no conflicts. Not blocking: LANE_TOOLING is a hand-kept copy of the mutation-affected.yml The other threads are all fixed at this head, so please resolve them: shards.failed rejection, absent-module branch, job timeout assertion, step timeout caps, test-scope rename, config test rename, upload if: always() and docs pointer to LANE_TOOLING. The step-order thread does not apply: a step moved before "Run mutants for" lands in the setup slice, which needs an integer timeout-minutes, so the guard still fails. Before merge, the pending Smoke Tests jobs need to finish. The 15-job weekly sweep also needs to finish inside its 30-minute budget. |
|
The full Ubuntu budget trial caught selector shard 8 and error shard 2 overrunning the mutant limit. Both failure envelopes were uploaded, and the report correctly failed as incomplete. Fixed and published at The real timeout artifacts also revealed that envelope-only shards must be read independently of reports. The actual CLI regression failed before the fix and now rejects either a failed envelope or a passing envelope without its report, even without All runnable gates passed on this exact head, including 52 harness tests. The revised full Ubuntu sweep is running; final-head CI will provide the remaining timing evidence. |
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
The weekly sweep at 3b1a22f still does not complete, so this needs one more change before merge. Run 38238796571 at this head failed on two shards. Each shard spends about 8m20s on the Stryker dry run before it mutates anything. After that, kernel-errors-2 (167 of 172 mutants) and selectors-2 (467 of 541) both hit the 22-minute step deadline. The shard split balances syntax nodes, not wall time, so the shards are uneven. The kernel-errors shards hold 116, 172 and 103 mutants. Selectors-2 needed more than 22 minutes while its siblings finished in 7 to 11. Every weekly run at this head would publish an incomplete-sweep failure and no score. That misses the goal of this PR. The rule to satisfy is that for every matrix entry from The remaining open thread, on the third kernel-errors shard, does not apply anymore, so you can resolve it: #3475 (comment). The third shard is justified. At 3b1a22f even the 172-mutant shard of that 3-way split times out, so going back to 2 shards would be worse. The Smoke Tests run still in progress is likely unrelated. This diff touches only the mutation scripts, workflows and config, and the docs. The failing check that matters here is the weekly run above. I did not reproduce the Stryker mutant counts or run the harness tests locally. I checked the changes by reading the code. |
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
|
The earlier findings at 3b1a22f are now fixed at 4db7adc: the native-count planner replaces the syntax-density one, and all 16 Mutants shards in weekly run 38244490637 finished green, the largest (kernel-errors-2) at about 21m20s for the whole job. I found no new problems in the code. The one thing still missing is proof that the merged sweep scores correctly: the "Mutation score report" job in that run (or a rerun at 4db7adc) must finish green, with its "Score the merged sweep" step passing --expect-shards equal to the 16-entry matrix. That job is still queued, and the only change since the green shards at 61a5b34 is docs. CI is pending. Smoke Tests are queued and in progress, and this diff does not touch what they exercise. Mutants (snapshot-occlusion) is queued and does run the changed harness, but it has not failed, and the same shard passed in the weekly run at the code-identical 61a5b34. I know of no conflicts. Once the report job passes, nothing else stops the merge. On the open inline threads from cubic-dev-ai: the cost thread #3475 (comment) does not apply, because the 8m20s is the Stryker initial test run and the planner only instruments sources, which takes under a second. The shard-count thread #3475 (comment) does not apply, because the 20m43s kernel-errors-2 measurement shows a 2-way split would overrun, so I did not reproduce the per-shard mutant counts or the 391/3,471 totals locally. I did not run the harness unit tests. I read the identity guard at sharding.ts:97 but did not run it, and I did not check that Stryker locations are 0-based, which the offset() mapping in sharding.ts assumes. A wrong base would throw the span error rather than lose mutants silently, and the CI shards passed. The deadline margin is thin, since kernel-errors-2 reached about 20m43s against 22m. Runner variance could push a future run over, and one green run does not show headroom. |
|
The full native-count Ubuntu sweep passed all 16 shards and aggregation: https://github.com/callstack/agent-device/actions/runs/38244490637. Its mutation code is identical to current head 4db7adc; only contributor guidance was shortened afterward. All 5,461 raw mutant identities are unique and every reported source matches the current checkout. Error shards took 15m38s/20m43s/9m35s; the heaviest selector shard took 19m58s. All 12 review threads are resolved. Every current-head PR check passed, including Coverage, Integration, and the complete canary (mutants: 15m05s): https://github.com/callstack/agent-device/actions/runs/38245625375. There is measured room for a follow-up:
These are temporary macOS experiments, not shipped changes or Ubuntu promises; an existing same-input broad report supplied the scope index. A one-file owner-only audit was rejected because it lost three runtime kills. Safe provenance/invalidation and a fresh broad reference remain necessary. The passing Ubuntu sweep spent 74 of 176 mutation-step runner-minutes on initial suites (42%). Halving that component would recover about 37 runner-minutes before index overhead. Prioritize coverage-derived scope and focused static coverage, then conservative incremental PR feedback; retain fresh weekly sweeps. Reports should expose fresh/reused work, static/hybrid status, covering tests, and uncovered decisions. The current report has 197 uncovered mutants (166 in selectors), providing a concrete strengthening backlog before broad inventory expansion. |
|
Holistic follow-up at An independent read-only native-instrumenter audit compared unsharded versus partitioned output by file, location, mutator, replacement, and ignore reason: kernel-errors 391→391 across three shards, selectors 3,471→3,471 across eight, with zero missing, extra, or duplicated identities. Stryker's public instrumenter creates a fresh collector per invocation, and its range API uses zero-based locations; the OXC/native boundary mapping is consistent. The pinned instrumenter avoids maintaining a second mutation-weight model. Reviewed the real CLI failure paths, scope discovery, scoring, workflow selection and budgets, dependency/lockfile scope, and the test ownership. Nonzero Stryker exits, missing/failed envelopes, report-less artifacts, missing module coverage, and short shard sets fail; available scores remain visible. Scope discovery does not execute tests, and per-test coverage remains enabled. The full 16-shard sweep and its aggregate passed, as did the final-head end-to-end canary. No source changes were needed in this follow-up. The remaining operational caveat is runtime margin: kernel-errors-2 took 20m43s against the 22-minute step deadline (77 seconds remaining). One passing sweep does not establish a long-term runtime guarantee. Coverage-derived scopes and measured cost packing remain the supported follow-up, rather than changing the production mutation universe or hiding failures. The new main commit since the passing CI changes recording overlays; it does not require rebasing this mutation-only diff. |
|
The missing evidence at 4db7adc is now in place. The "Mutation score report" job in weekly run 38244490637 passed, and its "Score the merged sweep" step ran with The 77-second margin on kernel-errors-2 against the 22-minute step deadline is not blocking, but a slower runner could tip it over, so the coverage-derived scope follow-up is worth tracking. |
|
Both PRs were rechecked. #3475 has no unresolved review threads. Testing the follow-up on the supported minimum Node 22.12 also exposed a parent test that incorrectly demanded warning-free stderr. Fixed at The production mutation code is unchanged, so the previous full 16-shard Ubuntu sweep remains applicable. Updated-head CI is running. #3506 was rebased onto this parent and its five findings were fixed, replied to and resolved; its 64 harness tests also pass on Node 22.12 with inherited environment state, including a real Stryker reporter loader smoke. |
0839f60 to
ba9cfec
Compare
|
Rebased onto main at Validated head |
|
Thanks for the update. At ba9cfec the code looks clean, and the earlier review at 4db7adc was also clean. The new commit only touches scripts/mutation/*.test.ts (imports and one assertion), so nothing in the harness, CI job definitions or docs changed since then. I did not re-run the harness tests or check:affected locally, so the pass result is yours. The earlier threads show open=0 and resolved=12, the same as at 4db7adc, so I read it as no thread changing since. CI is still pending: both Smoke Tests jobs are queued or running. This PR changes the mutation harness, CI job definitions and docs, and none of the daemon or device routes that smoke covers, so a smoke failure would most likely not come from it. No conflicts. Once Smoke Tests finish green on ba9cfec, the PR is ready for a human merge. |
|
The fresh Ubuntu run exposed a second deadline cause: The parent now uses five error shards, aligned with the weekly matrix and its 18-report completeness count. Native Stryker instrumentation preserves all 391 error mutant identities exactly once, with partitions 76/75/72/75/93 instead of 129/128/134. No test exclusions, source removals, score-policy changes or timeout extensions. This adds two dry runs in exchange for shorter mutation work per job and deadline margin. Both exact heads pass |
|
The PR is ready from the code side. The earlier review at ba9cfec found nothing blocking, and this update at ff85105 still finds nothing blocking. Not blocking: the weekly sweep now runs five shards of 76/75/72/75/93 mutants, and each shard still pays the roughly 8m15s dry run, so at about 8 s per mutant the 93-mutant shard 5/5 needs about 21 minutes against the 22-minute step limit. No CI run at this commit ran a kernel-errors shard, so that margin is an estimate. You can take it or leave it, but one dispatch of the weekly sweep (or forcing kernel-errors in the affected lane) would confirm that shard 5/5 finishes with room to spare. I did not reproduce the 391-mutant identity preservation or the partition sizes, and I did not run the harness tests locally, so those are your results. The diff touches only the mutation scripts, the GitHub Actions files and config, so it does not overlap the smoke route. Two Smoke Tests jobs are still pending, the other two passed, and all other checks pass, including the affected mutation lane. There are no conflicts. Once the two pending Smoke Tests jobs finish green, the PR is ready for human review. |
|
Fresh Ubuntu validation confirms the capacity fix at parent The current pilot run passed all 17 selected shards and aggregation: six modules, 4,995 unique raw identities, identical tested-merge provenance across reports and zero harness errors. These remain advisory mutation outcomes: 3,628 assertion kills, 1,122 survivors, 193 uncovered, 33 timeouts and 19 ignored. Coverage, Integration, all other main CI and all pilot platform checks passed. All review threads on both PRs are resolved. The parent’s final iOS smoke job is still running; its main CI and mutation canary/aggregation have passed. |
|
All CI is now green on both exact pushed heads: #3475 Both exact heads passed |
|
Final state refresh confirms both PRs were merged at 17:46 UTC. CI is green on both tested heads and every review thread is resolved. The earlier mergeability note described the open stack; the closed PRs now report stale/unknown mergeability. Main contains the five error shards and matching eighteen-report weekly completeness requirement. |
Summary
Weekly run 38114094784 exceeded deadlines and aggregated only 9/10 reports. Related discovery executed tests twice; partitions concentrated expensive mutation work.
21 files changed. Five error shards leave deadline margin after the observed 8-minute dry run.
Validation
ff85105bab046bab3df6f61bef076d987263a9b8:pnpm check:affected --runpassed every runnable gate; all 54 harness tests also passed on Node 22.12. Rebased ontod97c9bfbf3; use canonical test-kit fixtures after main’s helper migration.61a5b346d2(historical evidence before the test-kit rebase). Verified 5,461 unique raw identities and matching source. Slowest step: 20m43s. Current canary passed through aggregation: 390 unique identities, zero errors, five timeouts; 13m53s. All current CI passed, including Coverage/Integration and Apple smoke. Five error shards passed: slowest 16m57s. All review threads resolved.