Skip to content

fix(e2e): bake each fixture with npm ci from a committed lock - #311

Merged
ryanleecode merged 14 commits into
mainfrom
fix/e2e-locked-fixtures
Oct 12, 2026
Merged

ryanleecode merged 14 commits into
mainfrom
fix/e2e-locked-fixtures

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The e2e bake stopped resolving dependencies. Each fixture now installs with one npm ci from its committed package-lock.json, so an upstream npm publish can no longer stall or change the lane. Main's e2e has been stuck since @effect/platform-node@4.0.3 was published: it requires peer effect@^4.0.3, which does not exist, and a fresh resolution of ^4.0.0 picks it.

Plan: docs/plans/2026-10-11-1203-fix-e2e-locked-fixtures-plan.md.

What changes

  • bake-fixtures.sh runs one npm ci --no-audit --no-fund --loglevel=error per fixture. The two npm install calls, --min-release-age, timeout and kill wrapper are gone. On failure the script passes npm's output and exit code through unchanged and adds one line naming fixtures:lock.
  • The committed locks leave out each closure tarball's integrity and version. At bake time the harness copies version, bin, dependencies, optionalDependencies, peerDependencies and peerDependenciesMeta onto every closure entry from that tarball's freshly packed manifest (filledLockOf in @systemfsoftware/stryker-e2e-core). A source edit or a release therefore installs without regenerating any lock.
    • version and the edge fields are what npm ci validates (npm/lib/utils/validate-lockfile.js; arborist builds edges from those four fields in node.js loadDeps).
    • bin is there because npm links bins from the lock's metadata, not from the extracted tarball (arborist/lib/arborist/rebuild.js calls binLinks({ pkg: node.package })). Without it, a changed bin would install the stale link with exit 0.
  • pnpm pack --out <packs>/%s.tgz writes tarballs without a version in the file name. The staged fixture manifest lists the whole closure, aliases included, as file:../../packs/<name>.tgz devDependencies (ClosureInstall.dependencies).
  • pnpm --filter @systemfsoftware/stryker-e2e fixtures:lock (BakedFixtureCache.lockProgram) stages every fixture the way the bake does, runs npm install --package-lock-only --ignore-scripts, and writes the lock back with closure integrity and version removed.
  • The seven test/e2e/testResources/*/package-lock.json files are seeded from fix(repo): resolve the e2e fixture bake against a pinned registry snapshot #285's locks and regenerated by that script. Re-running it leaves all seven byte-identical (checked after the bin change too).

Done (c): the @effect pins equal pnpm-lock.yaml

pnpm-lock.yaml:
  effect@4.0.0
  @effect/platform-node@4.0.0
  @effect/platform-node-shared@4.0.0

fixture                          effect platform-node platform-node-shared
calc-fixture                     4.0.0 4.0.0 4.0.0
enterprise-monorepo-fixture      4.0.0 4.0.0 4.0.0
failing-fixture                  4.0.0 4.0.0 4.0.0
svelte-app-fixture               4.0.0 4.0.0 4.0.0
typescript-checker-fixture       4.0.0 4.0.0 4.0.0
vitest-nested-describe-fixture   4.0.0 4.0.0 4.0.0
vm-vitest-fixture                4.0.0 4.0.0 4.0.0

Each row is jq -r '[.packages["node_modules/effect"].version, .packages["node_modules/@effect/platform-node"].version, .packages["node_modules/@effect/platform-node-shared"].version]|join(" ")' over that fixture's lock.

No closure entry in any lock keeps integrity or version. jq '[.packages[]|select((.resolved//"")|startswith("file:"))|select(has("integrity") or has("version"))]|length' prints 0 for all seven.

Done (d): drift fails with npm's own error

This ran in a scratch copy outside the worktree, with no VM. calc-fixture was staged the way the bake stages it: catalog specs resolved, closure devDependencies written, lock filled from 17 freshly packed tarballs. The control npm ci exited 0 (added 149 packages in 1s), and node_modules/.bin/stryker pointed to ../@systemfsoftware/stryker-js/dist/main.mjs. Each case below then ran the bake script against that directory.

Lock deleted:

[bake] calc-fixture: npm ci
npm error code EUSAGE
npm error
npm error The `npm ci` command can only install with an existing package-lock.json or
npm error npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm@5 or
npm error later to generate a package-lock.json file, then try again.
...
[bake] calc-fixture: npm ci exited 1; if package-lock.json drifted or is missing, run pnpm --filter @systemfsoftware/stryker-e2e fixtures:lock and commit the result

effect bumped from ^4.0.0 to ^4.0.2 in the staged package.json:

[bake] calc-fixture: npm ci
npm error code EUSAGE
npm error
npm error `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm error
npm error Invalid: lock file's effect@4.0.0 does not satisfy effect@4.0.3
...
[bake] calc-fixture: npm ci exited 1; if package-lock.json drifted or is missing, run pnpm --filter @systemfsoftware/stryker-e2e fixtures:lock and commit the result

Both failed in under a second, with exit code 1 from npm.

Why npm hung instead of failing with ETARGET

The cause is unknown. In a scratch install, a packed tarball depending on @effect/platform-node@^4.0.0, installed with no lock, printed nothing and ignored SIGTERM until it was killed at 125 s. The same edges declared in the root manifest failed with ERESOLVE in 2 s.

Known gap: regeneration does not pin to pnpm-lock.yaml

The root ruled that a fixture lock and pnpm-lock.yaml must record the same version wherever both resolve a package. This PR does not enforce that yet.

  • fixtures:lock copies each fixture's directory, committed lock included, into staging (test/e2e/src/Harness/fixture-cache.service.ts:301, fs.copy). It then runs npm install --package-lock-only there (:738, LOCK_ONLY_INSTALL at :114). npm keeps every locked version that still satisfies its range, so the 4.0.0 pins survive only because fix(repo): resolve the e2e fixture bake against a pinned registry snapshot #285's locks carried them.
  • With a missing lock, a new fixture, or a closure member's range change, ^4.0.0 resolves fresh and picks @effect/platform-node@4.0.3, the same release that stalled the bake.
  • Of the 242 package names that both pnpm-lock.yaml and the seven locks resolve, 53 already differ in version (e.g. @redis/client 6.3.0 vs 6.2.1). None of them is one of the @effect packages above.
  • The proposed fix is a catalog-scoped pin. Regeneration would take each catalog name's version from pnpm-lock.yaml's catalogs.default section, which records one version per name, and would apply it the same way at bake time so npm ci still matches. The root has not yet decided whether that lands in this PR (as more commits on this branch) or in a follow-up.

Known risks

  • Install scripts. Closure entries keep hasInstallScript from the lock, and the fill-in does not refresh it. None of the 17 packed closure manifests has a preinstall, install or postinstall script (scanned with tar -xOzf <tgz> package/package.json), so nothing changes today.
  • committableLockOf has no unit test. Its output is the seven committed locks; the jq check above shows no closure entry keeps integrity or version.
  • withClosureDevDependencies has no unit test. Only the bake covers it, and only indirectly. In the scratch copy, a staged manifest without the closure devDependencies still passed npm ci (exit 0, added 39 packages), because npm prunes the lock's extra closure entries instead of refusing them. The fixture then has no node_modules/.bin/stryker, so the e2e legs fail at the first CLI run rather than at the bake.
  • A stalled npm ci has no bound. The plan and the root ruled out any timeout, kill wrapper or retry. The [bake] <fixture>: npm ci line printed before each install still names the fixture.
  • A fixture's own file: tarball dependency would break. The strip removes integrity and version from every file:*.tgz entry, but only closure tarballs are refilled. None of the seven locks has such an entry.
  • Tarball names can collide. Without the version, @x/y and x-y would both pack to x-y.tgz. That cannot happen while every closure member is @systemfsoftware-scoped.
  • The bake cache key does not cover harness code (filledLockOf, withClosureDevDependencies). A local baked entry can outlive a change to them, as before this PR; CI starts cold.

Tests

  • In-source property suite in test/e2e-core/src/closure-lock.ts. It sits in-source because the repo's test-discipline lint rejects a src/__tests__ property file for a module that is not a workflow.
    • ∀m_StaleClosureEntry_≡ManifestVersionDependenciesAndBin: a stale closure entry takes the manifest's version, dependencies and bin, and nothing else (the manifest's license must not be copied).
    • ∀d_DependencyAbsentFromLock_∈FilledEntry: a dependency the lock lacks lands in the filled entry.
    • ∀r_RegistryEntry_≡Unchanged: registry entries pass through unchanged.
  • Red runs:
    • Against a stub fill-in that returned the lock unchanged, the first two properties failed and the third passed.
    • Against the previous five-field list without bin, the stale-entry property was refuted (2 shrinks).
    • Neither red was committed.
  • pnpm --filter @systemfsoftware/stryker-e2e-core test at 670048b92: 45 files, 349 tests pass. After the bin change the property suite passes 3/3, and typecheck, oxlint and tsgo lint are clean for e2e-core. Typecheck and lint are also clean for e2e. pnpm format:check exits 0.
  • No local e2e or mutation run. The five e2e legs on this PR are the end-to-end proof.

Size

git diff --numstat origin/main...HEAD, excluding docs/plans and the package-lock.json files. 254 hand-written lines; the 3 pnpm-lock.yaml lines are generated.

File Added Removed
docs/solutions/build-errors/e2e-bake-resolved-an-npm-alias-edge-from-the-registry.md 3 3
docs/solutions/build-errors/e2e-lane-packed-a-subset-of-its-workspace-closure.md 4 4
pnpm-lock.yaml (generated tsx edge) 3 0
test/e2e-core/src/closure-lock.ts 117 0
test/e2e-core/src/install-closure.workflow.ts 12 5
test/e2e-core/src/mod.ts 1 0
test/e2e/AGENTS.md 1 1
test/e2e/README.md 2 2
test/e2e/package.json 2 0
test/e2e/scripts/lock-fixtures.ts 7 0
test/e2e/src/Harness/bake-key.schema.ts 0 12
test/e2e/src/Harness/fixture-cache.service.ts 99 64
test/e2e/tests/__fixtures__/bake-fixtures.sh 6 19
Total 257

Review notes

  • Simplify pass: done. Two reviewers reported, and every suggestion was line-neutral or added lines, so none was applied. The third reviewer was stopped.
  • Code review: lighter than the full reviewer set: one correctness read plus one local adversarial reviewer, since no outside-model reviewer was installed. Compound Packs were not assessed in that review.
    • P1, regeneration pinning: the "Known gap" above, deferred to the root.
    • P2, bin drift: fixed in bdff834ea.
    • Two P3 doc findings: fixed in 4696b2199.
  • Supersedes ci/e2e-postsubmit's edits to bake-fixtures.sh and the README's install paragraph.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Compound pack check (report-only, ruled by the sub-conductor)

Against the changed files: 7 rules pass, 2 fail, 22 don't apply. Both failures are declared here, not fixed:

  • schema-laws rich-type-over-foreign-encoded: test/e2e-core/src/closure-lock.ts reads npm lock fields as plain JSON with no decoded type. This is deliberate: the unit's contract rules out npm wire schemas (plan KTD1).
  • boundary-testing pin-dependency-semantics: the tarball naming of pnpm pack --out (fixture-cache.service.ts:126) and the lock fields npm ci reads (closure-lock.ts) have no dedicated test that runs pnpm or npm. Two things exercise them for real instead: the five e2e legs, which pack the closure and run npm ci against the filled locks, and the Done (d) scratch runs above. A pnpm or npm upgrade that changes either behaviour fails an e2e leg. No extra test was added, to stay within the line limit.

Simplify pass: done with reason. Two reviewers reported suggestions that kept or added lines, and the third was stopped. The code review was report-only and lighter than the full reviewer set.

Requirements-only plan for fix/e2e-locked-fixtures: one npm ci per fixture from a committed package-lock.json, closure entries overlaid from packed manifests, no age gate, timeout or second install. Answers the root P0's four questions with registry, stalled-log and npm ci probe evidence.
Adds the planning contract (KTD1-KTD7), units U1-U4, verification contract and definition of done to the brainstorm plan; R4 copies npm's manifest-sourced lock fields and R9 counts tests and docs, per the root's rulings.
Adds the pure closure-lock functions the bake and the lock script share:
filledLockOf copies npm's manifest-sourced lock fields onto each file:
tarball entry from its packed manifest, committableLockOf drops the
integrity and version a source change or release moves, and
withClosureDevDependencies lists the closure on the fixture root.
ClosureInstall gains the closure as a name-to-spec record.
The bake no longer resolves: it stages each fixture with the packed
closure as root devDependencies and the committed package-lock.json
filled from the freshly packed manifests, then runs one npm ci per
fixture with no timeout, budget or age gate. A drifted or missing lock
fails with npm ci's own output and exit code plus a line naming
fixtures:lock.

Closure tarballs pack under version-free names (pnpm pack --out), so a
release does not move the specs the locks record. The staging path is
exported as BakedFixtureCache.stageProgram for the lock script.
A closure entry is filled with version, which npm ci compares against
the ideal tree (npm/lib/utils/validate-lockfile.js), and the four fields
arborist turns into edges (dependencies, optionalDependencies,
peerDependencies, peerDependenciesMeta: @npmcli/arborist/lib/node.js),
so a new dependency fails npm ci with Missing ... from lock file.
Every other committed field is kept as written.
Covers AE3 (a stale closure entry takes the packed manifest's version
and dependencies, other fields kept), AE4's precondition (a dependency
the lock lacks lands in the filled entry), and a registry entry passing
through unchanged with its integrity.
pnpm --filter @systemfsoftware/stryker-e2e fixtures:lock runs
BakedFixtureCache.lockProgram: it stages every fixture the way the bake
does, runs npm install --package-lock-only over the filled committed lock
and writes it back with each closure tarball's integrity and version
stripped. The script is the harness entry point under tsx, which
test/e2e now declares.

The seven locks are regenerated from #285's, which pin effect,
@effect/platform-node, @effect/platform-node-shared and
@effect/opentelemetry at 4.0.0, as pnpm-lock.yaml does.
…ill-in

The sub-conductor ruling narrows R4 to the fields npm ci checks, the
test-discipline lint puts the property suite in-source, the lock loop
moves into the harness as lockProgram, and U4 drops the hang lesson.
npm links bins from the lock's package metadata, not from the extracted
tarball (arborist rebuild.js binLinks({ pkg: node.package })), so a
closure member that changed its bin installed with the stale link. The
fill-in now copies bin with the version and edge fields.
README names the drift npm ci reports (a fixture range or a closure
edge) and the bin the bake copies from the packed manifest. The lane's
AGENTS.md and the alias lesson's guard bullet now describe the npm ci
bake instead of the positional-spec install.

@ryanleecode ryanleecode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 4696b21: all 5 e2e legs, check, checker-parity and Nix green; up to date with main; 0 open threads; ~250 hand-written lines plus 7 committed fixture locks; no skips, disables, caps or force flags. Only red is the bench enterprise leg's setup-timed-out, which fails on every PR and leaves PRs with Stream G's bench-main-only.

@ryanleecode
ryanleecode merged commit ddeb67a into main Oct 12, 2026
26 of 27 checks passed
@ryanleecode
ryanleecode deleted the fix/e2e-locked-fixtures branch October 12, 2026 16:38
systemfsoftware-maker added a commit that referenced this pull request Oct 12, 2026
…ke's lock

Merging #311 put a committed package-lock.json in the enterprise
fixture. The bench copies that directory from side B, so npm install
started from the bake's lock: an npm probe kept its 86 registry entries
at their locked versions and dropped the 18 closure entries, which point
at the bake's ../../packs layout. Versions then came from the e2e lock
instead of each side's pnpm-lock overrides and --before cutoff, and
withheldPins no longer described the install.

The bench now removes the lock from its own copy before installing, so
the enterprise install is the one the A/A runs 38140987766 and
38143012318 measured. The fixture, its lock and the bake are untouched.
systemfsoftware-maker added a commit that referenced this pull request Oct 12, 2026
Main's bake (#311) runs one npm ci per testResources fixture from its
committed package-lock.json and fails the whole bake when one is missing.
The diff-scope fixture this branch adds had none, and its .gitignore kept
the lock out of git.

The lock is seeded from calc-fixture's, which declares the same three
devDependencies, and normalized with fixtures:lock, so it pins effect and
@effect/platform-node at 4.0.0 like the seven locks on main. A fresh
resolution would have floated them to 4.0.3. The fixture's .gitignore no
longer lists package-lock.json; the guest's two commits keep the lock
unchanged, so the A-to-B diff still holds only src/target.ts.

Plan: docs/plans/2026-10-09-1850-feat-mutant-quality-plan.md (L2).
systemfsoftware-maker added a commit that referenced this pull request Oct 12, 2026
…d fixture bakes, LPT planner

Takes #303's per-project lockfile closure digest, #311's npm-ci fixture
bakes from committed locks, #301's shard planner, and #293's lawful
planMutantTests generators.

- The incremental cache version moves to 7. Main and this branch each moved
  it from 5 to 6 for different layouts, so neither may read the other's
  file. runInputsDigestOf keeps #303's lockfile closure input, and the store
  key reaches it through the same function. #303's closure files, tests and
  oracle fixtures are byte-identical to main.
- incremental-reuse.integration.test.ts keeps this branch's version: #303's
  only edit there was the version number in the incrementalSources scenario
  this branch deletes.
- plan-request.schema.ts keeps #301's PlanShardsRefused and drops the two
  schemas this branch retired.
- The planMutantTests law file takes #293's lawfulSubsumptionOf and drops
  this branch's untestedOf, which fixed the same draw.
- The e2e closure install is per fixture, so only fixtures that request the
  S3 driver install it, and it carries #311's devDependency map. The harness
  stages each fixture's planned dependencies into its manifest, and the bake
  runs npm ci. fixtures:lock regenerated the locks: the verdict-store
  fixture gains its own lock, and the other seven gain the yaml dependency
  #303 added to the engine.
- The docs say a store is shared only between checkouts at the same
  absolute path, because mutant ids hash the absolute file path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants