Skip to content

feat(repo): reuse CompileError verdicts keyed on the checker's whole program - #228

Merged
kiro-systemf[bot] merged 20 commits into
mainfrom
stryker/compile-error-reuse
Oct 7, 2026
Merged

kiro-systemf[bot] merged 20 commits into
mainfrom
stryker/compile-error-reuse

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This layer goes on top of #227. An incremental run now reuses CompileError verdicts instead of type-checking every one again. On main, CompileError is an ephemeral status (Mutant.schema.ts:25), so every run re-checks 3,888 mutants: 3,547 in stryker-js, 238 in typescript-checker and 103 in vitest-runner. That re-check is the largest fixed cost per shard; run 37622289361 on db180fc3, with nothing to re-test, still spent about 100 s per shard on stryker-js.

Reuse key (operator rule, cycle 116)

A remembered CompileError is reused only when the reuse key matches. The key covers every input of the type check:

  • the mutant's identity and its replacement (existing key);
  • the content hash (sha256) of every file in the TS program the checker loaded (Program.getSourceFiles()). That includes lib .d.ts files and node_modules declarations, sorted by normalised path so enumeration order cannot matter;
  • every tsconfig the program was built from;
  • the TypeScript version, the checker plugin version, and the checker's options.

The checker computes this digest and serves it through a new digest RPC. The engine stamps it beside each CompileError it writes. A missing or different digest means the mutant is re-checked, and the refusal reason is programChanged. Hashing the current program needs the checker running, so this saves the per-mutant checks, not checker startup.

Stream

ReuseReported.reused counts reused CompileError verdicts. ReuseRefusals gains programChanged, added as an optional key so stream lines from earlier engines still decode. The schema version is unchanged.

Breaking (BREAK-1)

Every checker worker must answer the digest RPC. The plugin interface and stryker-js therefore take major bumps; the TypeScript checker and the cli-contract take minor bumps.

Validation

  • Failing before: packages/stryker-js/tests/compile-error-reuse.integration.test.ts was run in a scratch worktree on origin/main and failed 4 of 4 scenarios. For example, "an unchanged rerun reuses the CompileError verdicts and checks nothing" got secondRanNothing: 4 (expected 0) and secondReusedAll: false (expected true).
  • Passing after (re-run by me): both acceptance suites pass, now including the extends-chain and relocation cases.
    • Engine (compile-error-reuse.integration.test.ts): an unchanged rerun reuses every verdict with zero checks; editing a transitively imported .d.ts re-checks; changing tsconfig re-checks; editing an unrelated test file keeps every verdict.
    • Real TypeScript checker (program-digest.integration.test.ts): runs the same cases against the checker's program digest.
  • Key properties: identify-program.workflow.property.test.ts checks that a permuted file set gives the same key, and that changing the TS version, checker version, options, an added file, or the tsconfig moves the key.
  • Gates (re-run by me on the head): format and check:ci (112/112, 17/17) pass, the stryker-js and typescript-checker suites pass, and the changeset check passes with intents for all 4 packages. Verdict-Semantics: unchanged: a verdict is reused only when the inputs equal those of the check that produced it, so statuses equal a from-scratch run (EXACT).
  • Effect on main: none until a release. Main's Mutation run uses the published CLI, so the effect shows on the second main run after a release that contains this layer; the first run writes the digests.

Review (correctness, adversarial and simplify lenses, before opening)

Fixed, each with a test that fails before the fix:

  • P0: the tsconfig extends chain, both relative paths and package specifiers, is now hashed into the digest. The checker and engine suites each gain a test that edits only the base config and expects a re-check.
  • P1: digest paths are now relative to the project root. The checker runs inside .stryker-tmp/sandbox-<random>/, so with absolute paths the key would never match on CI. A new test copies the same fixture into two directories and gets the same digest.
  • Checker options are canonicalized. An encoding failure now produces no digest, so the mutant is re-checked, where before it produced an empty string.
  • A failed closure analysis now gets its own refusal reason.
  • Simplified: one key builder, one version reader, and program files are read with bounded concurrency.

Dismissed:

  • Hashing the raw tsconfig rather than the checker's override text. The override is a pure function of the raw text and the checker version, and both are in the key.
  • One checker failure dropping every digest. That only forces a re-check.
  • @noble/hashes. It is the repo convention, and the PLUG-1 build gate passes.

Nothing was run with stryker.

Borrowed / rejected

  • Borrowed: Bazel action keys hash every declared input, toolchain version and flags, not just the changed file. PIT history reuses results only when the class and its tests are unchanged.
  • Rejected: keying on the mutated file's import closure. The engine's closure analysis excludes .d.ts and node_modules, so that key would be too narrow.

…l file (#155)

A dry-run-only run with incremental mode on now writes its dry-run coverage into the incremental file, keeping any verdicts already there, so a later run that reads the file through incrementalSources skips its own dry run. The run-inputs digest no longer covers dryRunOnly, so a dry-run-only preflight and the runs that reuse its coverage share one digest; existing caches are refused once with runInputsChanged.

(cherry picked from eafe8f3)

Verdict-Semantics: unchanged
The TypeScript checker reports the digest of the program it loaded: every source
file the program holds, hashed, the tsconfig files it was built from, the
TypeScript version, the checker plugin version and the checker's options. The
engine stamps that digest beside every CompileError verdict it records, and a
later incremental run reuses such a verdict only when the digest it recomputes is
byte-equal. A missing digest, a changed file anywhere in the program, a changed
tsconfig, toolchain or option all send the mutant back to the checker. Hashing
the program needs the checker running, so the saving is the per-mutant check

Holding the digest means a plugin interface that can answer `digest`, and a
`programChanged` refusal reason on the reuse stream line. Verdicts the engine did
not remember keep their previous keys and refusals

Verdict-Semantics: unchanged
The reuse stream's refused counts gained a `programChanged` member, and the
generated contract document listed it as required, so a consumer built against the
committed stream would have refused every line an earlier engine wrote. The count
is optional, the way the plan line's `projects` array is: a stream line without it
still decodes

Verdict-Semantics: unchanged
Local builds render the Plugin namespace export as Node and CI renders Node_2 as Node (docs/solutions/api-extractor-node-alias-nondeterminism.md). #227 committed the local rendering and api:check failed in run 37626449206; main's rendering is the one CI accepts

Verdict-Semantics: unchanged
Every checker worker must answer the new digest RPC, so the plugin interface and the engine take a major bump per BREAK-1

Verdict-Semantics: unchanged
… CI and local builds

Run 37626449206 failed api:check on a report regenerated locally; run 37649609332 passed with main's rendering

Verdict-Semantics: unchanged
The digest covered the root tsconfig and its project references but not the
configs reached through "extends", so editing an extended base left the
program key unchanged and CompileError verdicts were reused across a real
program change. The chain is now followed the way TypeScript resolves it -
relative, absolute and node_modules package specifiers, string or array form -
and every file in it is hashed. An unresolvable or unparseable config fails the
digest, which forces a re-check. The engine-level scenario needs a fixture
checker that digests the same chain, which the program-digesting fixture now
does

Verdict-Semantics: unchanged
Every digest entry was an absolute path, so the checker's sandbox directory
(.stryker-tmp/sandbox-<random>) put a different key on the same program on
every run and CompileError verdicts were never reused on CI. Paths are now
named relative to the root tsconfig's directory; files outside it, such as
node_modules realpaths, become deterministic "../" paths

Verdict-Semantics: unchanged
The options JSON was re-encoded in insertion order, so semantically equal
option sets produced different keys and never matched; and a value that is not
JSON was silently replaced with {}, keying a configuration the checker never
ran. Object keys are now sorted before hashing and an unencodable option set
fails the digest, which forces a re-check

Verdict-Semantics: unchanged
A refusal caused only by a failed closure analysis was reported as
closureChanged, and a keyed CompileError whose program still matched was
reported the same way - neither names the input that was actually missing. The
stream now carries a closureAnalysisFailed count, and that reason is chosen
whenever the closure analysis failed and no earlier gate outranks it

Verdict-Semantics: unchanged
cacheKeyOf and programKeyOf differed only in the digest they read, so one
keyOf now serves both. readTypescriptVersion and readCheckerVersion share one
reader whose failure returns no version and so no digest. The program digest
borrows the existing sha256 and optional-field helpers instead of local
copies, and reads program files with bounded concurrency

Verdict-Semantics: unchanged
The reuse-refusal objects these scenarios compare gained a
closureAnalysisFailed count, so each zero-refusal fixture now names it

Verdict-Semantics: unchanged
The published stream schema gains the optional closureAnalysisFailed count
that the reuse report now emits alongside the other refusal counts

Verdict-Semantics: unchanged
The digest helpers now branch through Option and Boolean match, so no function
exceeds the complexity budget, the key-order canonicaliser sorts schema keys
without assertions, and the closure walker resolves extends candidates without
a loop. The optional-field builder moves to the reuse module as a dual export,
which keeps it out of a cell file and off the pipeable-signature rule

Verdict-Semantics: unchanged
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #229 October 7, 2026 17:17
@systemfsoftware-maker systemfsoftware-maker changed the title stryker/compile error reuse feat(repo): reuse CompileError verdicts keyed on the checker's whole program Oct 7, 2026
…ools now decode

pnpm-release-management main renamed versioning.strategy pnpm to changesets (#9, #33), and both the Changeset Check and Release callers run its main, so every pull request here failed with 'cannot parse config release.jsonc' (run 37657899053)

Verdict-Semantics: unchanged
Base automatically changed from stryker/dry-run-only-coverage to main October 7, 2026 18:07
…ners

Run 37664668214 timed out e2e sabotage and rest-1 at 19m59s with no failing test, on a tree identical to d82f758, which passed in run 37658299516

Verdict-Semantics: unchanged

@kiro-systemf kiro-systemf Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CompileError reuse keyed on every program file + tsconfigs + TS/checker versions; tests cover .d.ts and tsconfig changes. 8/8 green.

@kiro-systemf
kiro-systemf Bot merged commit 34f3a03 into main Oct 7, 2026
8 checks passed
@kiro-systemf
kiro-systemf Bot deleted the stryker/compile-error-reuse branch October 7, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant