Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
292 changes: 292 additions & 0 deletions docs/plans/2026-10-11-1203-fix-e2e-locked-fixtures-plan.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ A registry fallback for a workspace package is never a valid outcome. An optiona

### I3: Tarball lookup is exact

_A member's tarball is `<scope>-<name>-<semver>.tgz`, not any file that starts with `<scope>-<name>-`._ With prefix matching, `@systemfsoftware/stryker-js` could select `systemfsoftware-stryker-js-vitest-runner-*.tgz`. That would drop the CLI's own tarball from the bake cache key, and now that the install is planned from the looked-up tarballs, from the install too.
_A member's tarball is `<scope>-<name>.tgz`, packed by `pnpm pack --out`, not any file that starts with `<scope>-<name>-`._ With prefix matching, `@systemfsoftware/stryker-js` could select `systemfsoftware-stryker-js-vitest-runner-*.tgz`. That would drop the CLI's own tarball from the bake cache key and from the install. Without the version in the name, a release does not move the spec a committed fixture lock records.

`installClosure` and its refusals in `@systemfsoftware/stryker-e2e-core` implement I1 and I2. The harness plans the install only when a fixture needs baking, and passes the plan to `bake-fixtures.sh` as arguments. The step that copied the packed runner over the alias directory is deleted.
`installClosure` and its refusals in `@systemfsoftware/stryker-e2e-core` implement I1 and I2. The harness writes the plan into each staged fixture manifest as devDependencies, and `bake-fixtures.sh` runs one `npm ci` per fixture from its committed lock. The step that copied the packed runner over the alias directory is deleted.

## Anti-Pattern Code Smells

Expand All @@ -67,7 +67,7 @@ _A member's tarball is `<scope>-<name>-<semver>.tgz`, not any file that starts w

## Verification & Prevention

- `install-closure.integration.test.ts` in `@systemfsoftware/stryker-e2e-core` runs under `pnpm test`. It packs a closure with the production alias shape and installs the plan with `npm install --package-lock-only --offline`, an empty cache and an unreachable registry. It then requires `file:` provenance for every workspace package in the lockfile. Any registry fallback for a closure member fails the install. Its other scenarios cover each I2 refusal and the optional-peer exemption.
- `install-closure.integration.test.ts` in `@systemfsoftware/stryker-e2e-core` runs under `pnpm test`. It packs a closure with the production alias shape and installs the plan's positional specs (`ClosureInstall.specs`, which the bench lane still installs) with `npm install --package-lock-only --offline`, an empty cache and an unreachable registry. It then requires `file:` provenance for every workspace package in the lockfile. Its other scenarios cover each I2 refusal and the optional-peer exemption. The bake installs the same plan as staged devDependencies with `npm ci` from each fixture's committed lock; only the e2e legs exercise that path.
- Two-sided check (scratch evidence, not a committed fixture): the same offline install given the tarball list alone exits `ENOTCACHED` for `@systemfsoftware/stryker-js-vitest-runner`, and given the plan it exits 0.

## Related
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@ _What the lane packs and installs is computed from the manifests, never enumerat

_Each step of a multi-phase install owns exactly one failure class, and its message names it._ The step boundary is what makes a red lane self-diagnosing:

| Step | Signature | Owner |
| --------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `npm install the <fixture> registry dependencies` | `ERESOLVE` on the fixture's own pinned dev/runtime deps | the fixture's dependency pins — a test-matrix decision, e.g. a `^N` range whose publisher later peers a newer exact version than the fixture pins |
| `npm install the workspace closure tarballs in <fixture>` | `ETARGET` naming a `@systemfsoftware/stryker-*` package, or any failure resolving a closure member | this defect class (I1) |
| Step | Signature | Owner |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `plan the workspace closure install` | `UnpackedWorkspaceDependency` naming a `@systemfsoftware/stryker-*` package the closure did not pack | this defect class (I1) |
| `[bake] <fixture>: npm ci exited <status>` | npm ci's own `Missing: <name>@<version> from lock file`, `Invalid: …` or `EUSAGE` | a committed fixture lock that drifted from the closure: run `fixtures:lock` and commit the result |

## Anti-Pattern Code Smells

Expand Down
3 changes: 3 additions & 0 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

117 changes: 117 additions & 0 deletions test/e2e-core/src/closure-lock.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
import { Array, Function, Option, Predicate, Record } from 'effect'
import type { Json, JsonObject } from 'effect/Schema'

const EDGE_FIELDS = ['dependencies', 'optionalDependencies', 'peerDependencies', 'peerDependenciesMeta']
const FILLED_FIELDS = ['version', 'bin', ...EDGE_FIELDS]
const UNPINNED_FIELDS = ['integrity', 'version']
const CLOSURE_TARBALL = /^file:.*\/([^/]+\.tgz)$/

const objectOf = (json: Json | undefined): Option.Option<JsonObject> =>
Option.liftPredicate(json, (value): value is JsonObject => Predicate.isObject(value))

const mapClosureEntries = (lock: Json, rewrite: (entry: JsonObject, tarball: string) => JsonObject): Json =>
Option.getOrElse(
Option.gen(function*() {
const object = yield* objectOf(lock)
const packages = yield* objectOf(object['packages'])
return Object.assign({}, object, {
packages: Record.map(packages, (entry): Json =>
Option.getOrElse(
Option.gen(function*() {
const closureEntry = yield* objectOf(entry)
const resolved = yield* Option.liftPredicate(closureEntry['resolved'], Predicate.isString)
const tarball = yield* Array.get(CLOSURE_TARBALL.exec(resolved) ?? [], 1)
return rewrite(closureEntry, tarball)
}),
() => entry,
)),
})
}),
() => lock,
)

export const filledLockOf: {
(manifests: Readonly<Record<string, Json>>): (lock: Json) => Json
(lock: Json, manifests: Readonly<Record<string, Json>>): Json
} = Function.dual(
2,
(lock: Json, manifests: Readonly<Record<string, Json>>): Json =>
mapClosureEntries(lock, (entry, tarball) =>
Option.match(Option.flatMap(Record.get(manifests, tarball), objectOf), {
onNone: () => entry,
onSome: (manifest) =>
Object.assign(
Record.filter(entry, (_, key) => !FILLED_FIELDS.includes(key)),
Record.filter(manifest, (_, key) => FILLED_FIELDS.includes(key)),
),
})),
)

export const committableLockOf = (lock: Json): Json =>
mapClosureEntries(lock, (entry) => Record.filter(entry, (_, key) => !UNPINNED_FIELDS.includes(key)))

export const withClosureDevDependencies: {
(closure: Readonly<Record<string, string>>): (manifest: Json) => Json
(manifest: Json, closure: Readonly<Record<string, string>>): Json
} = Function.dual(
2,
(manifest: Json, closure: Readonly<Record<string, string>>): Json =>
Option.match(objectOf(manifest), {
onNone: () => manifest,
onSome: (object) =>
Object.assign({}, object, {
devDependencies: Object.assign({}, Option.getOrUndefined(objectOf(object['devDependencies'])), closure),
}),
}),
)

if (import.meta.vitest !== void 0) {
const { it } = await import('@systemfsoftware/vitest')
const Equal = await import('effect/Equal')
const S = await import('effect/Schema')

const tarball = 'systemfsoftware-stryker-js.tgz'
const resolved = `file:../../packs/${tarball}`
const bin = { stryker: 'dist/main.mjs' }
const lockOf = (entry: JsonObject): Json => ({ packages: { 'node_modules/@systemfsoftware/stryker-js': entry } })

it.prop(
'∀m_StaleClosureEntry_≡ManifestVersionDependenciesAndBin',
{ of: [S.NonEmptyString, S.NonEmptyString], subject: filledLockOf },
(subject, [name, version]) => {
const stale = {
resolved,
dev: true,
bin,
version: `${version}-stale`,
dependencies: { [`dep-${name}-stale`]: '1' },
}
const manifest = { version, license: 'MIT', bin: { [name]: name }, dependencies: { [`dep-${name}`]: version } }
const filled = { resolved, dev: true, bin: manifest.bin, version, dependencies: manifest.dependencies }
return Equal.equals(subject(lockOf(stale), { [tarball]: manifest }), lockOf(filled))
},
)

it.prop(
'∀d_DependencyAbsentFromLock_∈FilledEntry',
{ of: [S.NonEmptyString, S.NonEmptyString], subject: filledLockOf },
(subject, [name, version]) => {
const stale = { resolved, dev: true, dependencies: { effect: '^4.0.0' } }
const manifest = { version, dependencies: { effect: '^4.0.0', [`dep-${name}`]: version } }
const filled = { resolved, dev: true, version, dependencies: manifest.dependencies }
return Equal.equals(subject(lockOf(stale), { [tarball]: manifest }), lockOf(filled))
},
)

it.prop(
'∀r_RegistryEntry_≡Unchanged',
{ of: [S.NonEmptyString, S.NonEmptyString], subject: filledLockOf },
(subject, [name, version]) => {
const lock = lockOf({ version, resolved: `https://registry.npmjs.org/${name}.tgz`, integrity: `sha512-${name}` })
return Equal.equals(
subject(lock, { [tarball]: { version: '0.0.0', dependencies: { [`dep-${name}`]: '1' } } }),
lock,
)
},
)
}
17 changes: 12 additions & 5 deletions test/e2e-core/src/install-closure.workflow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ type ClosureInstallTypeId = typeof ClosureInstallTypeId

export class ClosureInstall extends S.TaggedClass<ClosureInstall>()('ClosureInstall', {
specs: S.Array(S.String),
dependencies: S.Record(S.String, S.String),
}) {
readonly [ClosureInstallTypeId] = ClosureInstallTypeId
}
Expand Down Expand Up @@ -186,16 +187,22 @@ const conflictRefusal = (
},
)

const aliasSpecOf = (command: InstallClosureCommand, edge: ClosureEdge): ReadonlyArray<string> =>
Option.toArray(Option.map(tarballOf(command, edge.target), (path) => `${edge.dependency}@file:${path}`))
const aliasTargetOf = (command: InstallClosureCommand, edge: ClosureEdge): ReadonlyArray<readonly [string, string]> =>
Option.toArray(Option.map(tarballOf(command, edge.target), (path) => [edge.dependency, path] as const))

const installOf = (command: InstallClosureCommand, aliases: ReadonlyArray<ClosureEdge>): ClosureInstall =>
ClosureInstall.make({
const installOf = (command: InstallClosureCommand, aliases: ReadonlyArray<ClosureEdge>): ClosureInstall => {
const aliasTargets = aliases.flatMap((edge) => aliasTargetOf(command, edge))
return ClosureInstall.make({
specs: [
...command.members.map((member) => member.tarballPath),
...Arr.dedupe(aliases.flatMap((edge) => aliasSpecOf(command, edge))).sort(),
...Arr.dedupe(aliasTargets.map(([dependency, path]) => `${dependency}@file:${path}`)).sort(),
],
dependencies: Object.fromEntries(
[...command.members.map((member) => [member.manifest.name, member.tarballPath] as const), ...aliasTargets]
.map(([name, path]) => [name, `file:${path}`]),
),
})
}

const decide = (command: InstallClosureCommand): Result.Result<ClosureInstall, InstallClosureFailure> => {
const edges = command.members.flatMap((member) => memberEdgesOf(member.manifest))
Expand Down
1 change: 1 addition & 0 deletions test/e2e-core/src/mod.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ export * from './bench-run.schema.js'
export * from './bench-summary.schema.js'
export * from './catalog-resolution.js'
export * from './catalog-resolution.schema.js'
export * from './closure-lock.js'
export * from './closure.schema.js'
export * from './confirm-annotations.workflow.js'
export * from './covering-test-files.workflow.js'
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# AGENTS.md — `@systemfsoftware/stryker-e2e`

Private E2E lane for the shipped `stryker` artifact: it packs the workspace closure, bakes every fixture's
`npm install` in a preparation microVM into a content-addressed host cache, boots one warm
`npm ci` from its committed lock in a preparation microVM into a content-addressed host cache, boots one warm
`@systemfsoftware/effect-microsandbox` microVM per test file and fixture on the digest-pinned `node:24-alpine` image,
copies the baked fixture onto its disk at `/work`, snapshots it, and runs each engine invocation in a fresh
copy-on-write fork of that snapshot. Publishes no artifact. Parent: `test/AGENTS.md`.
Expand Down
4 changes: 2 additions & 2 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ The suite resolves the workspace closure of the CLI and its plugins, packs each
- Full mutation runs against realistic test suites
- Exit codes and typed machine-mode JSON error envelopes on failure

The closure installs in one `npm install` with no `@systemfsoftware` workspace package taken from the registry. `pnpm pack` rewrites a workspace alias (`"@systemfsoftware/stryker-js-vm-runner": "workspace:@systemfsoftware/stryker-js-vitest-runner@^"`) to `npm:<target>@^<version>`, and npm resolves an `npm:` spec from the registry even when the target's tarball is in the same install. So the install passes every packed tarball plus one `<alias>@file:<target tarball>` spec per alias edge between closure members (`installClosure` in `@systemfsoftware/stryker-e2e-core`). Global setup fails, naming the edge, instead of reaching the registry when a packed member depends on a workspace package the closure did not pack (`UnpackedWorkspaceDependency`), when one alias name would install two different packages (`ConflictingAliasTargets`), or when a fixture manifest names a workspace package itself (`FixtureNamesWorkspacePackage`). The closure is installed into every fixture, so a fixture never names it.
Each fixture installs with one `npm ci` from its committed `package-lock.json`, with no `@systemfsoftware` workspace package taken from the registry. `pnpm pack` rewrites a workspace alias (`"@systemfsoftware/stryker-js-vm-runner": "workspace:@systemfsoftware/stryker-js-vitest-runner@^"`) to `npm:<target>@^<version>`, and npm resolves an `npm:` spec from the registry even when the target's tarball is in the same install. So the staged fixture manifest lists every packed tarball plus one `<alias>: file:<target tarball>` entry per alias edge between closure members as devDependencies (`installClosure` in `@systemfsoftware/stryker-e2e-core`). Global setup fails, naming the edge, instead of reaching the registry when a packed member depends on a workspace package the closure did not pack (`UnpackedWorkspaceDependency`), when one alias name would install two different packages (`ConflictingAliasTargets`), or when a fixture manifest names a workspace package itself (`FixtureNamesWorkspacePackage`). The closure is installed into every fixture, so a fixture never names it.

Both bake installs resolve registry dependencies fresh, so they pass `--min-release-age=1`: a release less than a day old never reaches a fixture. A half-published release, such as `@effect/platform-node@4.0.3` declaring a peer `effect@^4.0.3` that was not yet on the registry, otherwise leaves npm resolving forever. Each install runs under a 300 s `timeout`, with a KILL 10 s later for an npm that ignores TERM. When one runs over, the bake fails and its stderr, which global setup reports, names the fixture and the step that ran over. The guest's `timeout` is busybox, which exits with the killed command's status (143 or 137) rather than GNU's 124, so the script decides "ran over" from elapsed time, not from the exit code.
The committed locks pin every registry dependency, so an upstream publish never changes what a fixture installs. They omit each closure tarball's `integrity` and `version`, which a source change or release moves; the bake fills those, the tarball's `bin` and its dependency fields back in from the freshly packed manifests, so a changed bin needs no regeneration. When a fixture's dependency range or a closure tarball's dependency edges no longer match the lock, or the lock is missing, the bake fails with `npm ci`'s own error and a line naming the fix: run `pnpm --filter @systemfsoftware/stryker-e2e fixtures:lock` and commit the result.

## Running locally

Expand Down
2 changes: 2 additions & 0 deletions test/e2e/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
},
"type": "module",
"scripts": {
"fixtures:lock": "tsx --conditions=@systemfsoftware/source scripts/lock-fixtures.ts",
"lint": "f=${OXLINT_FORMAT:-${AGENT:+agent}}; oxlint . --format=${f:-default}",
"seed-status-fault": "node scripts/seed-status-fault.ts",
"test:e2e": "vitest run",
Expand All @@ -37,6 +38,7 @@
"effect": "catalog:",
"microsandbox": "catalog:",
"oxlint": "catalog:",
"tsx": "catalog:",
"typescript": "catalog:",
"vitest": "catalog:"
},
Expand Down
7 changes: 7 additions & 0 deletions test/e2e/scripts/lock-fixtures.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import { runMain } from '@effect/platform-node/NodeRuntime'
import { layer } from '@effect/platform-node/NodeServices'
import { Effect } from 'effect'

import { BakedFixtureCache } from '../src/Harness/fixture-cache.service.js'

runMain(BakedFixtureCache.lockProgram.pipe(Effect.provide(layer)))
12 changes: 0 additions & 12 deletions test/e2e/src/Harness/bake-key.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,12 @@ import { Schema } from 'effect'

export const PackedPackage = Schema.Struct({
name: Schema.String,
version: Schema.String,
fileName: Schema.String,
tarballPath: Schema.String,
})

export type PackedPackage = typeof PackedPackage.Type

export class FoundPackage extends Schema.TaggedClass<FoundPackage>()('Found', { pack: PackedPackage }) {}

export class MissingTarball extends Schema.TaggedClass<MissingTarball>()('MissingTarball', {
prefix: Schema.String,
directory: Schema.String,
}) {}

export const PackedPackageLookup = Schema.Union([FoundPackage, MissingTarball])

export type PackedPackageLookup = typeof PackedPackageLookup.Type

export const TurboTask = Schema.Struct({
command: Schema.optional(Schema.String),
package: Schema.optional(Schema.String),
Expand Down
Loading
Loading