diff --git a/.gitignore b/.gitignore index a76f7ce..83ddaf3 100644 --- a/.gitignore +++ b/.gitignore @@ -8,5 +8,5 @@ experiments/**/output/ app-icon.png # Tauri regenerates these schemas on every build examples/**/src-tauri/gen/ -# the CLI's default test-root directory (release-qa designate/status use it when --root is not given) +# the CLI's default test root and run state (.release-qa, .release-qa/runs) when --root/--state are not given .release-qa/ diff --git a/README.md b/README.md index 7b7452c..9debb61 100644 --- a/README.md +++ b/README.md @@ -12,35 +12,42 @@ The first targets are Tauri applications on Windows and Linux, with Dot X as the ## Status Stage 0 (proving the assumptions) is complete. The runner (environment checks, scenario execution, the durable run -journal) and a first slice of the CLI (`doctor`, `designate`, `status`) exist; running an actual scenario from the -CLI (`run`, `resume`), the Tauri driver, the dashboard and the GitHub integration do not yet. +journal) and the CLI's local commands (`doctor`, `designate`, `status`, `reset`, `run`, `resume`) exist. The Tauri +driver adapter and a runnable sample consumer, the dashboard and the GitHub integration do not exist yet. - [Native automation](docs/decisions/native-automation.md): unchanged packaged Tauri apps can be driven on Windows and Ubuntu. The Dot X feasibility check is still open. - [GitHub merge gate](docs/decisions/github-gate.md): a no-service required check works, with documented design changes and unproven items. - [Tool layout and defaults](docs/decisions/tool-layout.md): runtime, package manager, baselines and repository structure. +- [Local runs](docs/decisions/local-runs.md): the local candidate manifest, run state, resume rules and exit codes. ## CLI -Run it straight from a checkout; no build step (see [tool layout](docs/decisions/tool-layout.md)). `designate` and -`status` work as they stand, against any directory: +Run it straight from a checkout; no build step (see [tool layout](docs/decisions/tool-layout.md)): ```sh node packages/qa/src/cli/main.ts designate [--root ] [--json] # marks a directory safe to install and delete into node packages/qa/src/cli/main.ts status [--root ] [--json] # reports what a designated root holds, dirty or clean +node packages/qa/src/cli/main.ts reset [--root ] [--json] # reaps what a crashed run left, clears the dirty marker +node packages/qa/src/cli/main.ts doctor --project --profile [--json] +node packages/qa/src/cli/main.ts run --project --candidate --profile --suite [--root ] [--state ] [--json] +node packages/qa/src/cli/main.ts resume --run [--state ] [--json] ``` -`doctor` needs a `qa/project.json` from a project that has one (this repository does not ship a sample yet — that -lands with the CLI's `run`/`resume` commands): - -```sh -node packages/qa/src/cli/main.ts doctor --project path/to/qa/project.json --profile windows [--json] -``` - -`--root` defaults to `.release-qa` under the current directory (gitignored) when not given. Every command prints to -stdout on success and to stderr on failure; `--json` switches both to one line of machine-readable JSON. Exit codes: -`0` passed/ready, `1` a scenario the candidate failed (not reachable yet — no command runs a scenario), `2` this -machine does not meet a requested profile, `3` anything else that stopped the command (bad usage, an unreadable or -invalid project file, a profile the project does not declare). +`doctor` and `run` need a consumer's `qa/project.json`; this repository does not ship a runnable sample consumer yet. +`run` also needs a [local candidate manifest](docs/decisions/local-runs.md#the-local-candidate-manifest) naming the +file to test and its SHA-256, which is checked before anything is installed. + +`--root` defaults to `.release-qa` and `--state` to `.release-qa/runs`, under the current directory (gitignored). +Results go to stdout, problems and progress to stderr; `--json` makes the result one line of JSON. `run` prints its +run id on stderr before anything runs, so an interrupted run can be continued with `resume`. Interrupting `run` once +(Ctrl+C) cancels it cleanly; a second interrupt exits at once. + +Exit codes: `0` passed/ready; `1` a scenario the candidate failed; `2` a missing prerequisite or manual work left (this +machine does not meet the profile, a scenario was blocked, a manual check remains); `3` anything else that stopped the +command or left a run unfinished (bad usage, a file that cannot be read or does not verify, an unknown profile or +suite, an interrupted or cancelled scenario, a cleanup that failed and left the environment dirty, a reset that could not +clean everything). A run takes the highest rule +that applies: any failure is `1` even if something else was also interrupted. ## Development diff --git a/docs/decisions/local-runs.md b/docs/decisions/local-runs.md new file mode 100644 index 0000000..506fbe8 --- /dev/null +++ b/docs/decisions/local-runs.md @@ -0,0 +1,58 @@ +# Decision: running a suite from a checkout (Task 2.2, part 3a) + +Status: **implemented** for the CLI's `run`, `resume` and `reset`, against fixture consumer projects. Running the real sample (`examples/tauri-smoke`) through a Tauri driver adapter is part 3b. + +## The local candidate manifest + +`run --candidate ` takes a local manifest, not a GitHub candidate record: + +```json +{ + "schemaVersion": 1, + "id": "local-2026-09-23.1", + "artifacts": [ + { "profile": "windows", "name": "setup.exe", "path": "dist/setup.exe", "sha256": "<64 lower-case hex>" } + ] +} +``` + +- One artifact per profile; `path` is relative to the manifest's own directory and cannot leave it, lexically or through a link: the real location must be inside the manifest's real directory. +- Before anything is installed, the chosen profile's file is hashed at its real location and must match `sha256`; the location is re-checked after hashing. Other profiles' files are never read. +- It has no source or build provenance, so it can never stand in for a GitHub candidate at the merge gate (Stage 3). Hooks receive only `candidate.id` and the verified `artifact` (`name`, its real absolute `path` with no link left in it, `sha256`); a full GitHub candidate record also fits `candidate`. + +## Consumer code + +`qa/project.json` names a lifecycle module exporting `lifecycle` (`install`, `reset`, `launch`, `cleanup`) and scenario files exporting `scenarios` (`{ id, setup?, steps }[]`). A requirement key `/` runs the scenario with that id. Running executes the project's code; the CLI does it only for a project the user points it at, and every problem (missing hook, undefined scenario, duplicate id, a module that throws while loading) is reported before anything is installed. + +## State + +- Default test root: `.release-qa` under the current directory; default state directory: `.release-qa/runs` (both gitignored). Each run is `runs//` with `invocation.json` (what was asked, absolute paths, and the tested artifact's SHA-256), `events.jsonl` (the Task 1.3 journal) and `summary.json`. +- The machine is identified in run records by a random token kept in the state directory, never the host name. +- `run` announces `run started` on stderr before anything runs, in every output mode, with the `resume` command to use (including a custom `--state`, quoted so it pastes safely in bash and PowerShell), so a run can be resumed even if the process dies. +- A run id is a run id, never a path: `resume --run` accepts only the journal's id grammar, which has no path separators (`/`, `\`, `:`). + +## Journal and resume + +Each scenario records a `scenario-started` checkpoint, then an attempt, then a `cleanup-failed` checkpoint if its cleanup failed. `resume --run `: + +- re-verifies the candidate: the same manifest id, and bytes whose SHA-256 is the one recorded when the run started (a manifest edited to name new bytes under the same id is refused: that is a different build); +- carries **passed** and **failed** forward (a failure cannot disappear by being run again), with any recorded cleanup failure; +- reruns anything else (blocked, cancelled, interrupted, never reached) as a retry (`retryOf`) of its latest attempt; +- turns a `scenario-started` with no attempt after it (the process died) into an explicit `interrupted` attempt first, so the crash stays in the run's history. + +A run whose journal has conflicting or cyclic events, or cannot be read, is not continued. + +## Outcomes and exit codes + +One scenario failing does not stop the others; cancellation stops the running scenario (its cleanup still runs) and starts nothing further (`not-run`). Exit codes, highest rule first: any **failed** → `1`; any interrupted, cancelled or not-run, or any cleanup that failed (the environment was left dirty, however the scenario went) → `3`; any blocked or manual → `2`; otherwise `0`. Configuration and verification problems are `3` before anything runs. + +## Reset + +A crash can leave owned resources and a dirty marker in the test root, which blocks later runs. `reset` reaps the ledger and clears the marker only when that fully succeeds; it refuses an undesignated root and a root a run currently holds. + +## Not verified + +- Cancellation by a real signal is tested on Linux only (CI). On Windows a console Ctrl+C reaches the same handler, but a test cannot send one to another process. +- Evidence files (screenshots, logs) are not collected yet; attempts record `evidence: []`. +- The artifact's contents could still change after verification; only copying it into the run's own storage would close that. Links cannot redirect it, and the install hook consumes it at once. +- Consumer code runs in the CLI's own process: a scenario that calls `process.exit` takes the CLI with it. That is the crash `resume` recovers from, not something prevented. diff --git a/packages/qa/src/cli/args.ts b/packages/qa/src/cli/args.ts index ed0ab67..f6aa3e9 100644 --- a/packages/qa/src/cli/args.ts +++ b/packages/qa/src/cli/args.ts @@ -25,7 +25,34 @@ export interface StatusCommand { json: boolean; } -export type Command = DoctorCommand | DesignateCommand | StatusCommand; +export interface ResetCommand { + name: 'reset'; + root: string | undefined; + json: boolean; +} + +export interface RunCommand { + name: 'run'; + project: string; + candidate: string; + profile: string; + suite: string; + /** The designated test root; absent means the caller's default. */ + root: string | undefined; + /** Where run journals are kept; absent means the caller's default. */ + state: string | undefined; + json: boolean; +} + +/** Continues a run exactly as it was started: the project, candidate, profile, suite and root are the run's own. */ +export interface ResumeCommand { + name: 'resume'; + run: string; + state: string | undefined; + json: boolean; +} + +export type Command = DoctorCommand | DesignateCommand | StatusCommand | ResetCommand | RunCommand | ResumeCommand; export type ParsedArgs = { ok: true; command: Command } | { ok: false; error: string; json: boolean }; @@ -40,6 +67,9 @@ const SPECS: Record = { doctor: { required: ['project', 'profile'], optional: [] }, designate: { required: [], optional: ['root'] }, status: { required: [], optional: ['root'] }, + reset: { required: [], optional: ['root'] }, + run: { required: ['project', 'candidate', 'profile', 'suite'], optional: ['root', 'state'] }, + resume: { required: ['run'], optional: ['state'] }, }; const COMMAND_NAMES = Object.keys(SPECS) as Command['name'][]; @@ -59,11 +89,24 @@ export function parseArgs(argv: readonly string[]): ParsedArgs { return { ok: true, command: { name, project: flags.values.project as string, profile: flags.values.profile as string, json } }; case 'designate': case 'status': - return { ok: true, command: { name, root: flags.values.root as string | undefined, json } }; + case 'reset': + return { ok: true, command: { name, root: flags.values.root, json } }; + case 'run': { + const { project, candidate, profile, suite, root, state } = flags.values as Record; + return { ok: true, command: { name, project: project!, candidate: candidate!, profile: profile!, suite: suite!, root, state, json } }; + } + case 'resume': { + // The run id becomes a directory name under the state directory, so it must never be a path. + const run = flags.values.run as string; + if (!RUN_ID.test(run)) return fail(`--run must be a run id (letters, digits, ".", "_" or "-", starting with a letter or digit), not ${JSON.stringify(run)}`, json); + return { ok: true, command: { name, run, state: flags.values.state, json } }; + } } } const countJson = (argv: readonly string[]): number => argv.filter((token) => token === '--json').length; +/** The id grammar the journal uses; it has no path separators, so a run id can never climb out of the state directory. */ +const RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; function isCommandName(value: string): value is Command['name'] { return (COMMAND_NAMES as string[]).includes(value); diff --git a/packages/qa/src/cli/candidate.ts b/packages/qa/src/cli/candidate.ts new file mode 100644 index 0000000..dd92894 --- /dev/null +++ b/packages/qa/src/cli/candidate.ts @@ -0,0 +1,81 @@ +import { createHash } from 'node:crypto'; +import { createReadStream } from 'node:fs'; +import { readFile, realpath, stat } from 'node:fs/promises'; +import { dirname, resolve, sep } from 'node:path'; +import { parseLocalCandidate } from '../model/local-candidate.ts'; +import type { ArtifactRef, CandidateRef } from '../runner/execute.ts'; + +export type LoadedCandidate = { ok: true; candidate: CandidateRef; artifact: ArtifactRef } | { ok: false; error: string }; + +const message = (error: unknown): string => (error instanceof Error ? error.message : String(error)); +const fold = (path: string): string => (process.platform === 'win32' ? path.toLowerCase() : path); +const isInside = (directory: string, path: string): boolean => fold(path).startsWith(fold(directory.endsWith(sep) ? directory : directory + sep)); + +/** + * Reads a local candidate manifest, picks the artifact for `profile`, and checks the file's bytes against the + * manifest's SHA-256 before anything is installed. Never throws. Only the chosen profile's file is read. + */ +export async function loadCandidate(manifestPath: string, profile: string): Promise { + let text: string; + try { + text = await readFile(manifestPath, 'utf8'); + } catch (error) { + return { ok: false, error: `could not read the candidate manifest ${manifestPath}: ${message(error)}` }; + } + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch (error) { + return { ok: false, error: `${manifestPath} is not valid JSON: ${message(error)}` }; + } + const result = parseLocalCandidate(parsed); + if (!result.ok) return { ok: false, error: `${manifestPath}: ${result.error.message}` }; + const manifest = result.value; + + const chosen = manifest.artifacts.find((a) => a.profile === profile); + if (chosen === undefined) { + return { ok: false, error: `candidate "${manifest.id}" has no artifact for profile "${profile}"; it has: ${manifest.artifacts.map((a) => a.profile).join(', ')}` }; + } + + // Relative to the manifest, not to wherever the command happens to be run from. + const path = resolve(dirname(manifestPath), ...chosen.path.split('/')); + const info = await stat(path).catch(() => undefined); + if (info === undefined) return { ok: false, error: `the artifact ${path} for profile "${profile}" does not exist` }; + if (!info.isFile()) return { ok: false, error: `the artifact ${path} for profile "${profile}" is not a file` }; + // The manifest's path check is lexical; a link on the way could still lead elsewhere. Judge the real locations, then + // hash the real file and give hooks that path, so no link is left in it that could be swapped to point elsewhere. + let realDirectory: string; + let realFile: string; + try { + [realDirectory, realFile] = await Promise.all([realpath(dirname(manifestPath)), realpath(path)]); + } catch (error) { + return { ok: false, error: `could not resolve the artifact ${path}: ${message(error)}` }; + } + if (!isInside(realDirectory, realFile)) { + return { ok: false, error: `the artifact ${path} for profile "${profile}" leads outside the manifest's directory, to ${realFile}` }; + } + + let actual: string; + try { + actual = await sha256Of(realFile); + // A link swapped while the file was being hashed would make the check above describe some other file. + if ((await realpath(path)) !== realFile) return { ok: false, error: `the artifact ${path} changed location while it was being verified` }; + } catch (error) { + return { ok: false, error: `could not read the artifact ${path}: ${message(error)}` }; + } + if (actual !== chosen.sha256) { + return { ok: false, error: `the artifact ${path} does not match the candidate: expected SHA-256 ${chosen.sha256}, found ${actual}` }; + } + return { ok: true, candidate: { id: manifest.id }, artifact: { name: chosen.name, path: realFile, sha256: actual } }; +} + +/** Streams the file, so an installer of any size is hashed without being held in memory. */ +function sha256Of(path: string): Promise { + return new Promise((resolveHash, rejectHash) => { + const hash = createHash('sha256'); + createReadStream(path) + .on('data', (chunk) => hash.update(chunk)) + .on('error', rejectHash) + .on('end', () => resolveHash(hash.digest('hex'))); + }); +} diff --git a/packages/qa/src/cli/consumer.ts b/packages/qa/src/cli/consumer.ts new file mode 100644 index 0000000..eebacd4 --- /dev/null +++ b/packages/qa/src/cli/consumer.ts @@ -0,0 +1,91 @@ +import { dirname, resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import type { Project } from '../model/project.ts'; +import type { Requirement } from '../model/requirement.ts'; +import type { Lifecycle, RunContext, Scenario } from '../runner/execute.ts'; + +export type LoadedConsumer = { ok: true; lifecycle: Lifecycle; scenarios: Scenario[] } | { ok: false; error: string }; + +interface ScenarioDefinition { + id: string; + setup?(ctx: RunContext): Promise; + steps(ctx: RunContext): Promise; +} + +const HOOKS = ['install', 'reset', 'launch', 'cleanup'] as const; +const message = (error: unknown): string => (error instanceof Error ? error.message : String(error)); + +/** + * Imports the project's lifecycle module and scenario files and binds each automated requirement to the scenario + * whose id is the part of its key after the profile. This executes the project's code: the caller runs it only for + * a project the user pointed the CLI at. Every problem is found before anything is installed. Never throws. + */ +export async function loadConsumer(projectPath: string, project: Project, requirements: readonly Requirement[]): Promise { + // Inspecting what a module exports runs its code too (a getter can throw), so all of it is inside this boundary. + try { + return await inspectConsumer(projectPath, project, requirements); + } catch (error) { + return { ok: false, error: `could not read the project's lifecycle or scenarios: ${message(error)}` }; + } +} + +async function inspectConsumer(projectPath: string, project: Project, requirements: readonly Requirement[]): Promise { + const base = dirname(resolve(projectPath)); + const load = async (relative: string): Promise<{ ok: true; module: Record } | { ok: false; error: string }> => { + const path = resolve(base, ...relative.split('/')); + try { + return { ok: true, module: (await import(pathToFileURL(path).href)) as Record }; + } catch (error) { + return { ok: false, error: `could not load ${path}: ${message(error)}` }; + } + }; + + const lifecycleModule = await load(project.lifecycleModule); + if (!lifecycleModule.ok) return lifecycleModule; + const lifecycle = lifecycleModule.module.lifecycle as Record | undefined; + const missingHooks = HOOKS.filter((hook) => typeof lifecycle?.[hook] !== 'function'); + if (missingHooks.length > 0) { + return { ok: false, error: `${project.lifecycleModule} must export \`lifecycle\` with ${HOOKS.join(', ')}; missing: ${missingHooks.join(', ')}` }; + } + + const definitions = new Map(); + for (const file of project.scenarioFiles) { + const loaded = await load(file); + if (!loaded.ok) return loaded; + const exported = loaded.module.scenarios; + if (!Array.isArray(exported)) return { ok: false, error: `${file} must export \`scenarios\`, an array of { id, setup?, steps }` }; + for (const [index, value] of exported.entries()) { + const definition = value as Partial | null; + if (typeof definition?.id !== 'string' || typeof definition.steps !== 'function' || (definition.setup !== undefined && typeof definition.setup !== 'function')) { + return { ok: false, error: `${file}: scenarios[${index}] must have a string id, a steps function and optionally a setup function` }; + } + if (definitions.has(definition.id)) return { ok: false, error: `scenario "${definition.id}" is defined more than once` }; + definitions.set(definition.id, definition as ScenarioDefinition); + } + } + + const scenarios: Scenario[] = []; + const undefinedIds: string[] = []; + for (const requirement of requirements) { + const id = requirement.key.slice(requirement.key.indexOf('/') + 1); + const definition = definitions.get(id); + if (definition === undefined) { + undefinedIds.push(requirement.key); + continue; + } + scenarios.push({ + id, + requirement, + steps: (ctx) => definition.steps(ctx), + ...(definition.setup === undefined ? {} : { setup: (ctx: RunContext) => (definition.setup as NonNullable)(ctx) }), + }); + } + if (undefinedIds.length > 0) return { ok: false, error: `no scenario file defines: ${undefinedIds.join(', ')}` }; + + const hooks = lifecycle as unknown as Lifecycle; + return { + ok: true, + scenarios, + lifecycle: { install: (ctx) => hooks.install(ctx), reset: (ctx) => hooks.reset(ctx), launch: (ctx) => hooks.launch(ctx), cleanup: (ctx) => hooks.cleanup(ctx) }, + }; +} diff --git a/packages/qa/src/cli/environment-commands.ts b/packages/qa/src/cli/environment-commands.ts index 70917b8..040f69f 100644 --- a/packages/qa/src/cli/environment-commands.ts +++ b/packages/qa/src/cli/environment-commands.ts @@ -1,4 +1,13 @@ -import { checkTestRoot, designateTestRoot, readDirty, readLedger, type OwnedResource, type TestRootCheck } from '../runner/resources.ts'; +import { + acquireTestRoot, + checkTestRoot, + designateTestRoot, + readDirty, + readLedger, + resetDirtyEnvironment, + type OwnedResource, + type TestRootCheck, +} from '../runner/resources.ts'; const message = (error: unknown): string => (error instanceof Error ? error.message : String(error)); @@ -48,3 +57,31 @@ export async function runStatus(root: string, options: EnvironmentCommandOptions return { ok: false, error: `could not read the state of ${root}: ${message(error)}` }; } } + +export type ResetResult = { ok: true; root: string; failures: string[] } | { ok: false; error: string }; + +/** + * Reaps what earlier runs left owned in this root and, only when that fully succeeds, clears the dirty marker so the + * root can be used again. Only a designated root is touched, and only while no run holds it: resetting under a run + * would reap resources that run still uses. Never throws. + */ +export async function runReset(root: string, options: EnvironmentCommandOptions = {}): Promise { + try { + const check = await checkTestRoot(root, options); + if (!check.ok) return { ok: false, error: `${root} is not a designated test root (${check.reason}); nothing was touched` }; + const lock = await acquireTestRoot(check.root); + if (!lock.ok) return { ok: false, error: `${check.root} is in use by ${lock.heldBy}; nothing was touched` }; + try { + const { failures } = await resetDirtyEnvironment(check.root); + return { + ok: true, + root: check.root, + failures: failures.map((f) => `${f.resource.label}: ${f.reason}${f.detail === undefined ? '' : ` (${f.detail})`}`), + }; + } finally { + await lock.release(); + } + } catch (error) { + return { ok: false, error: `could not reset ${root}: ${message(error)}` }; + } +} diff --git a/packages/qa/src/cli/main.ts b/packages/qa/src/cli/main.ts index fa0c2d5..7004742 100644 --- a/packages/qa/src/cli/main.ts +++ b/packages/qa/src/cli/main.ts @@ -1,15 +1,18 @@ -import { join } from 'node:path'; +import { join, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; +import type { ValidationIssue } from '../model/validate.ts'; +import type { ScenarioEvent } from '../runner/execute.ts'; import { parseArgs } from './args.ts'; import { runDoctor, type DoctorReport } from './doctor.ts'; -import { runDesignate, runStatus, type StatusReport } from './environment-commands.ts'; +import { runDesignate, runReset, runStatus, type StatusReport } from './environment-commands.ts'; import { loadProject } from './project.ts'; -import type { ValidationIssue } from '../model/validate.ts'; +import { resumeRun, startRun, type RunOptions, type RunSummary } from './run.ts'; /** - * `0` passed. `1` a scenario the candidate failed (not yet reachable: no command runs a scenario in this build). - * `2` a missing prerequisite the tester, not the tool, must resolve (e.g. this machine does not match a profile). - * `3` everything else that stops the command: bad usage, a file that cannot be read, an unknown profile. + * `0` passed. `1` a scenario the candidate failed. `2` a missing prerequisite or manual work the tester, not the + * tool, must resolve (this machine does not match a profile, a scenario was blocked, a manual check is left). + * `3` everything else that stops the command or leaves a run unfinished: bad usage, a file that cannot be read or + * does not verify, an unknown profile, an interrupted or cancelled scenario, an environment reset that failed. */ export const EXIT = { ok: 0, scenarioFailure: 1, missingPrerequisite: 2, infrastructure: 3 } as const; @@ -20,8 +23,16 @@ export interface Io { const defaultIo: Io = { log: (line) => console.log(line), error: (line) => console.error(line) }; -/** Runs one CLI invocation and returns the process exit code; never throws and never touches `process` itself. */ -export async function main(argv: readonly string[], io: Io = defaultIo, cwd: () => string = () => process.cwd()): Promise { +/** + * Runs one CLI invocation and returns the process exit code; never throws and never touches `process` itself. + * `signal` cancels a run in progress: the running scenario stops, is cleaned up, and nothing further starts. + */ +export async function main( + argv: readonly string[], + io: Io = defaultIo, + cwd: () => string = () => process.cwd(), + signal: AbortSignal = new AbortController().signal, +): Promise { const parsed = parseArgs(argv); if (!parsed.ok) { reportError(io, parsed.json, parsed.error); @@ -67,11 +78,83 @@ export async function main(argv: readonly string[], io: Io = defaultIo, cwd: () printStatus(io, command.json, result.report); return EXIT.ok; } + + case 'reset': { + const result = await runReset(command.root === undefined ? defaultRoot(cwd) : resolve(cwd(), command.root)); + if (!result.ok) { + reportError(io, command.json, result.error); + return EXIT.infrastructure; + } + if (command.json) io.log(JSON.stringify(result)); + else if (result.failures.length === 0) io.log(`reset ${result.root}`); + else io.error([`could not reset ${result.root}; it stays dirty:`, ...result.failures.map((f) => ` ${f}`)].join('\n')); + return result.failures.length === 0 ? EXIT.ok : EXIT.infrastructure; + } + + case 'run': + case 'resume': { + const stateDir = command.state === undefined ? join(defaultRoot(cwd), 'runs') : resolve(cwd(), command.state); + const customState = command.state === undefined ? undefined : stateDir; + const options: RunOptions = { + stateDir, + signal, + // Announced before anything runs, on stderr in every mode: if the process dies, this is how to resume it. + onStart: (runId: string) => io.error(`run ${runId} started; if it is interrupted, continue it with: ${resumeHint(runId, customState)}`), + // Progress goes to stderr, so stdout carries only the summary. + ...(command.json ? {} : { onEvent: (event: ScenarioEvent) => io.error(`${event.scenario}: ${event.phase} ${event.status}${event.detail === undefined ? '' : ` (${event.detail})`}`) }), + }; + const result = + command.name === 'run' + ? await startRun( + { + project: resolve(cwd(), command.project), + candidate: resolve(cwd(), command.candidate), + profile: command.profile, + suite: command.suite, + root: command.root === undefined ? defaultRoot(cwd) : resolve(cwd(), command.root), + }, + options, + ) + : await resumeRun(command.run, options); + if (!result.ok) { + reportError(io, command.json, result.error); + return EXIT.infrastructure; + } + printRun(io, command.json, result.summary); + return result.summary.exitCode; + } } } const defaultRoot = (cwd: () => string): string => join(cwd(), '.release-qa'); +/** + * The command that continues a run. With a custom state directory it must name it, and pasting it must not expand + * anything: a plain path goes in as it is, anything else in single quotes (literal in both bash and PowerShell), and a + * path that itself contains a single quote is named in prose rather than put into a command it would break. + */ +export function resumeHint(runId: string, stateDir: string | undefined): string { + const command = `resume --run ${runId}`; + if (stateDir === undefined) return command; + if (/^[A-Za-z0-9_.:\\/-]+$/.test(stateDir)) return `${command} --state ${stateDir}`; + if (!stateDir.includes("'")) return `${command} --state '${stateDir}'`; + return `${command} --state , which is: ${stateDir}`; +} + +function printRun(io: Io, json: boolean, summary: RunSummary): void { + if (json) { + io.log(JSON.stringify(summary)); + return; + } + io.log( + [ + `run ${summary.runId} (candidate ${summary.candidateId}, profile ${summary.profile}, suite ${summary.suite})`, + ...summary.results.map((r) => ` ${r.requirement}: ${r.outcome}${r.carried === true ? ' (from earlier)' : ''}${r.detail === undefined ? '' : ` - ${r.detail}`}`), + ...summary.results.filter((r) => r.cleanup !== undefined && !r.cleanup.ok).map((r) => ` cleanup after ${r.requirement} failed: ${r.cleanup?.failures.join('; ')}`), + ].join('\n'), + ); +} + /** `issues` is always present in JSON output, empty when there are none, so a consumer can key on it unconditionally. */ function reportError(io: Io, json: boolean, error: string, issues?: readonly ValidationIssue[]): void { io.error(json ? JSON.stringify({ ok: false, error, issues: issues ?? [] }) : error); @@ -114,7 +197,23 @@ function printStatus(io: Io, json: boolean, report: StatusReport): void { // Runs only when this file is the process's entry point (`node packages/qa/src/cli/main.ts ...`), never when a test // imports it as a module. if (process.argv[1] !== undefined && pathToFileURL(process.argv[1]).href === import.meta.url) { - main(process.argv.slice(2)).then((code) => { - process.exitCode = code; - }); + // The first interrupt cancels: the running scenario stops and its cleanup runs. A second one exits at once; the + // journal and the test root's ledger and dirty marker still say what was left, for `reset` and `resume`. + const controller = new AbortController(); + const interrupt = (): void => { + if (controller.signal.aborted) process.exit(EXIT.infrastructure); + controller.abort(); + }; + process.on('SIGINT', interrupt); + process.on('SIGTERM', interrupt); + main(process.argv.slice(2), defaultIo, () => process.cwd(), controller.signal).then( + (code) => { + process.exitCode = code; + }, + // main never rejects by design; if a bug makes it, report it as an infrastructure error, not a bare stack trace. + (error: unknown) => { + console.error(`release-qa stopped unexpectedly: ${error instanceof Error ? error.message : String(error)}`); + process.exitCode = EXIT.infrastructure; + }, + ); } diff --git a/packages/qa/src/cli/plan.ts b/packages/qa/src/cli/plan.ts new file mode 100644 index 0000000..24d290a --- /dev/null +++ b/packages/qa/src/cli/plan.ts @@ -0,0 +1,23 @@ +import type { EnvironmentProfile, Project } from '../model/project.ts'; +import { profileOf, type Requirement } from '../model/requirement.ts'; + +export type Plan = + | { ok: true; profile: EnvironmentProfile; automated: Requirement[]; manual: Requirement[] } + | { ok: false; error: string }; + +/** + * What one run covers: the suite's requirements for this profile, in suite order. Manual requirements are listed + * but never run; they are work left for a person. A suite with nothing for the profile is refused rather than + * reported as an empty pass. + */ +export function selectPlan(project: Project, profileId: string, suiteId: string): Plan { + const profile = project.profiles.find((p) => p.id === profileId); + if (profile === undefined) return { ok: false, error: `profile "${profileId}" is not defined by this project; known profiles: ${project.profiles.map((p) => p.id).join(', ')}` }; + const suite = project.suites.find((s) => s.id === suiteId); + if (suite === undefined) return { ok: false, error: `suite "${suiteId}" is not defined by this project; known suites: ${project.suites.map((s) => s.id).join(', ') || '(none)'}` }; + + const byKey = new Map(project.requirements.map((r) => [r.key, r])); + const selected = suite.requirements.filter((key) => profileOf(key) === profileId).flatMap((key) => byKey.get(key) ?? []); + if (selected.length === 0) return { ok: false, error: `suite "${suiteId}" has nothing for profile "${profileId}"` }; + return { ok: true, profile, automated: selected.filter((r) => r.mode === 'automated'), manual: selected.filter((r) => r.mode === 'manual') }; +} diff --git a/packages/qa/src/cli/run.ts b/packages/qa/src/cli/run.ts new file mode 100644 index 0000000..e8cc9f0 --- /dev/null +++ b/packages/qa/src/cli/run.ts @@ -0,0 +1,327 @@ +import { randomBytes, randomUUID } from 'node:crypto'; +import { mkdir, readFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import type { RequirementKey } from '../model/requirement.ts'; +import type { Attempt, Outcome } from '../model/result.ts'; +import { Collector, parseVersioned, type FieldSpec } from '../model/validate.ts'; +import type { EnvironmentProbes } from '../runner/environment.ts'; +import type { RunEvent } from '../runner/events.ts'; +import { executeScenario, type ExecutionContext, type ScenarioEvent } from '../runner/execute.ts'; +import { appendEvent, readRun, writeFileAtomic, writeSummary, type RunState } from '../runner/journal.ts'; +import { loadCandidate } from './candidate.ts'; +import { loadConsumer } from './consumer.ts'; +import { selectPlan } from './plan.ts'; +import { loadProject } from './project.ts'; + +/** What `run` was asked to do, kept next to the run so `resume` repeats exactly that. Paths are absolute. */ +export interface RunInvocation { + project: string; + candidate: string; + profile: string; + suite: string; + root: string; +} + +/** + * `manual`: work for a person, never run. `not-run`: the run was cancelled before reaching it; `resume` runs it. + * Everything else is the recorded attempt's outcome. + */ +export type ResultOutcome = Outcome | 'manual' | 'not-run'; + +export interface RequirementResult { + requirement: RequirementKey; + outcome: ResultOutcome; + /** The attempt that decided this result. */ + attempt?: string; + /** Recorded by an earlier session of this run and not rerun. */ + carried?: boolean; + reason?: string; + detail?: string; + cleanup?: { ok: boolean; failures: string[] }; +} + +export interface RunSummary { + runId: string; + candidateId: string; + profile: string; + suite: string; + results: RequirementResult[]; + exitCode: number; +} + +export type RunResult = { ok: true; summary: RunSummary } | { ok: false; error: string }; + +export interface RunOptions { + /** Where each run's journal lives, one directory per run. */ + stateDir: string; + signal: AbortSignal; + /** Progress, for display. */ + onEvent?: (event: ScenarioEvent) => void; + /** Called once the run is recorded and before anything runs, so its id is known even if the process then dies. */ + onStart?: (runId: string) => void; + /** Test seams. */ + probes?: EnvironmentProbes; + timeouts?: ExecutionContext['timeouts']; +} + +const INVOCATION_FILE = 'invocation.json'; +const MACHINE_FILE = 'machine-id'; +const STARTED = 'scenario-started'; +/** Recorded after an attempt whose cleanup failed, so a resumed run still reports it for the carried result. */ +const CLEANUP_FAILED = 'cleanup-failed'; + +/** + * Starts a run. Everything that can be checked without touching the machine is checked first (project, plan, + * candidate bytes, the consumer's code), so a mistake is reported before any run state exists or anything is + * installed. Never throws. + */ +export async function startRun(input: RunInvocation, options: RunOptions): Promise { + const invocation: RunInvocation = { ...input, project: resolve(input.project), candidate: resolve(input.candidate), root: resolve(input.root) }; + const prepared = await prepare(invocation); + if (!prepared.ok) return prepared; + + const runId = newRunId(); + const runDir = join(options.stateDir, runId); + try { + await mkdir(runDir, { recursive: true }); + // The artifact's digest is kept with the invocation, so resume can tell a rebuild under the same candidate id. + await writeFileAtomic(join(runDir, INVOCATION_FILE), `${JSON.stringify({ schemaVersion: 1, ...invocation, artifactSha256: prepared.artifact.sha256 }, null, 2)}\n`); + const journal = new Journal(runDir, runId, undefined, 0); + await journal.append('run-started', { runId, candidateId: prepared.candidate.id, profile: invocation.profile, machineId: await machineId(options.stateDir) }); + options.onStart?.(runId); + return { ok: true, summary: await execute(prepared, invocation, runId, journal, emptyState(), options) }; + } catch (error) { + return { ok: false, error: `run ${runId} stopped: ${message(error)}` }; + } +} + +/** + * Continues a run. Passed and failed results carry forward; a failure cannot disappear by being run again. Anything + * else (blocked, cancelled, interrupted, never reached) runs again as a retry of its latest attempt. A scenario that + * was started but never recorded, because the process died, is first recorded as interrupted, so the crash stays in + * the run's history. The candidate must still be the same bytes under the same identity. Never throws. + */ +export async function resumeRun(runId: string, options: RunOptions): Promise { + const runDir = join(options.stateDir, runId); + try { + const invocation = await readInvocation(runDir); + if (!invocation.ok) return invocation; + + const state = await readRun(runDir); + const start = state.events.find((e) => e.type === 'run-started'); + if (start === undefined || start.type !== 'run-started') return { ok: false, error: `run ${runId} has no recorded start` }; + if (state.conflicts.length > 0 || state.cyclic.length > 0) { + return { ok: false, error: `run ${runId}'s journal is inconsistent (conflicting or cyclic events); it cannot be continued safely` }; + } + + const prepared = await prepare(invocation.value); + if (!prepared.ok) return prepared; + if (prepared.candidate.id !== start.data.candidateId) { + return { ok: false, error: `run ${runId} tested candidate "${start.data.candidateId}", but the manifest now names "${prepared.candidate.id}"` }; + } + // The manifest could have been edited to name new bytes under the same id: that is a different build. + if (prepared.artifact.sha256 !== invocation.artifactSha256) { + return { ok: false, error: `run ${runId} tested an artifact with SHA-256 ${invocation.artifactSha256}, but the manifest now names ${prepared.artifact.sha256}: a different build` }; + } + + const last = state.events.at(-1); + const journal = new Journal(runDir, runId, last?.id, state.events.length); + return { ok: true, summary: await execute(prepared, invocation.value, runId, journal, state, options) }; + } catch (error) { + return { ok: false, error: `run ${runId} stopped: ${message(error)}` }; + } +} + +type Prepared = Extract>, { ok: true }>; + +async function prepare(invocation: RunInvocation) { + const loaded = await loadProject(invocation.project); + if (!loaded.ok) return { ok: false as const, error: loaded.error }; + const plan = selectPlan(loaded.project, invocation.profile, invocation.suite); + if (!plan.ok) return plan; + const candidate = await loadCandidate(invocation.candidate, invocation.profile); + if (!candidate.ok) return candidate; + const consumer = await loadConsumer(invocation.project, loaded.project, plan.automated); + if (!consumer.ok) return consumer; + return { ok: true as const, plan, candidate: candidate.candidate, artifact: candidate.artifact, consumer }; +} + +async function execute(prepared: Prepared, invocation: RunInvocation, runId: string, journal: Journal, state: RunState, options: RunOptions): Promise { + const results: RequirementResult[] = []; + let attemptCount = state.attempts.length; + const newAttemptId = (): string => `${runId}.a${++attemptCount}`; + + for (const scenario of prepared.consumer.scenarios) { + const key = scenario.requirement.key; + const history = historyOf(state, key); + + // The process died after starting this scenario and before recording it: say so before anything else. + let latest = history.latest; + if (history.startedAfterLatest) { + const interrupted: Attempt = { id: newAttemptId(), requirement: key, outcome: 'interrupted', evidence: [], ...(latest === undefined ? {} : { retryOf: latest.id }) }; + await journal.append('attempt-recorded', { attempt: interrupted }); + latest = interrupted; + } + + if (latest !== undefined && (latest.outcome === 'passed' || latest.outcome === 'failed')) { + results.push({ + requirement: key, + outcome: latest.outcome, + attempt: latest.id, + carried: true, + // The environment it left may since have been reset, but the attempt did leave it dirty; that stays on record. + ...(history.cleanupFailed ? { cleanup: { ok: false, failures: ['cleanup failed after this attempt, in an earlier session of this run'] } } : {}), + }); + continue; + } + if (options.signal.aborted) { + results.push({ requirement: key, outcome: 'not-run' }); + continue; + } + + await journal.append('checkpoint', { name: STARTED, requirement: key }); + const result = await executeScenario( + { + candidate: prepared.candidate, + artifact: prepared.artifact, + profile: prepared.plan.profile, + testRoot: invocation.root, + signal: options.signal, + emit: (event) => options.onEvent?.(event), + lifecycle: prepared.consumer.lifecycle, + ...(options.probes === undefined ? {} : { probes: options.probes }), + ...(options.timeouts === undefined ? {} : { timeouts: options.timeouts }), + }, + scenario, + ); + const attempt: Attempt = { id: newAttemptId(), requirement: key, outcome: result.outcome, evidence: [], ...(latest === undefined ? {} : { retryOf: latest.id }) }; + await journal.append('attempt-recorded', { attempt }); + if (!result.cleanup.ok) await journal.append('checkpoint', { name: CLEANUP_FAILED, requirement: key }); + results.push({ + requirement: key, + outcome: result.outcome, + attempt: attempt.id, + ...(result.reason === undefined ? {} : { reason: result.reason }), + ...(result.detail === undefined ? {} : { detail: result.detail }), + cleanup: result.cleanup, + }); + } + + for (const requirement of prepared.plan.manual) results.push({ requirement: requirement.key, outcome: 'manual' }); + await writeSummary(journal.runDir, await readRun(journal.runDir)); + return { runId, candidateId: prepared.candidate.id, profile: invocation.profile, suite: invocation.suite, results, exitCode: exitCodeOf(results) }; +} + +/** + * Any failure outranks everything: it is a verdict on the candidate. Then anything unfinished, or a cleanup that + * failed (the environment is left dirty, however the scenario went), then work left for a person. + */ +export function exitCodeOf(results: readonly RequirementResult[]): number { + const has = (...outcomes: ResultOutcome[]) => results.some((r) => outcomes.includes(r.outcome)); + if (has('failed')) return 1; + if (has('interrupted', 'cancelled', 'not-run') || results.some((r) => r.cleanup?.ok === false)) return 3; + if (has('blocked', 'manual')) return 2; + return 0; +} + +/** + * The latest recorded attempt for a requirement; whether a start was recorded after it with no result (a crash); and + * whether its cleanup was recorded as failed. + */ +function historyOf(state: RunState, key: RequirementKey): { latest?: Attempt; startedAfterLatest: boolean; cleanupFailed: boolean } { + let latest: Attempt | undefined; + let startedAfterLatest = false; + let cleanupFailed = false; + for (const event of state.events) { + if (event.type === 'attempt-recorded' && event.data.attempt.requirement === key) { + latest = event.data.attempt; + startedAfterLatest = false; + cleanupFailed = false; + } else if (event.type === 'checkpoint' && event.data.requirement === key) { + if (event.data.name === STARTED) startedAfterLatest = true; + if (event.data.name === CLEANUP_FAILED) cleanupFailed = true; + } + } + return { ...(latest === undefined ? {} : { latest }), startedAfterLatest, cleanupFailed }; +} + +/** Appends events in a chain: each names the one before it, so their order never depends on clocks. */ +class Journal { + readonly runDir: string; + private readonly runId: string; + private previous: string | undefined; + private count: number; + + constructor(runDir: string, runId: string, previous: string | undefined, count: number) { + this.runDir = runDir; + this.runId = runId; + this.previous = previous; + this.count = count; + } + + async append(type: T, data: Extract['data']): Promise { + const id = `${this.runId}.e${++this.count}`; + const event = { schemaVersion: 1, id, recordedAt: timestamp(), type, data, ...(this.previous === undefined ? {} : { prev: this.previous }) } as RunEvent; + const appended = await appendEvent(this.runDir, event); + if (!appended.ok) throw appended.error; + this.previous = id; + } +} + +const INVOCATION_SPEC: FieldSpec = { required: ['project', 'candidate', 'profile', 'suite', 'root', 'artifactSha256'] }; + +async function readInvocation(runDir: string): Promise<{ ok: true; value: RunInvocation; artifactSha256: string } | { ok: false; error: string }> { + let text: string; + try { + text = await readFile(join(runDir, INVOCATION_FILE), 'utf8'); + } catch { + return { ok: false, error: `there is no run at ${runDir}` }; + } + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch (error) { + return { ok: false, error: `${join(runDir, INVOCATION_FILE)} is not valid JSON: ${message(error)}` }; + } + const result = parseVersioned(parsed, INVOCATION_SPEC, (c: Collector, rec) => ({ + value: { + project: c.text(rec.project, 'project', { max: 4096 }), + candidate: c.text(rec.candidate, 'candidate', { max: 4096 }), + profile: c.profileId(rec.profile, 'profile'), + suite: c.id(rec.suite, 'suite'), + root: c.text(rec.root, 'root', { max: 4096 }), + } as RunInvocation, + artifactSha256: c.sha256(rec.artifactSha256, 'artifactSha256') as string, + })); + return result.ok ? { ok: true, ...result.value } : { ok: false, error: `${join(runDir, INVOCATION_FILE)}: ${result.error.message}` }; +} + +/** + * Identifies this machine in run records without saying anything about it: a random token kept in the state + * directory, never the host name, which uploads must not carry. + */ +async function machineId(stateDir: string): Promise { + const path = join(stateDir, MACHINE_FILE); + const existing = (await readFile(path, 'utf8').catch(() => '')).trim(); + if (existing !== '') return existing; + const created = `machine-${randomUUID()}`; + await mkdir(stateDir, { recursive: true }); + await writeFileAtomic(path, `${created}\n`); + return created; +} + +function newRunId(): string { + const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\.\d{3}Z$/, 'Z'); + return `run-${stamp}-${randomBytes(3).toString('hex')}`; +} + +const timestamp = (): string => new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'); +const message = (error: unknown): string => (error instanceof Error ? error.message : String(error)); + +function emptyState(): RunState { + return { + exists: false, events: [], attempts: [], truncated: null, corrupt: [], conflicts: [], missingPredecessors: [], cyclic: [], + pending: [], synced: [], ackMismatches: [], orphanAcks: [], missingEvidence: [], + }; +} + diff --git a/packages/qa/src/index.ts b/packages/qa/src/index.ts index f70822b..01a0ec1 100644 --- a/packages/qa/src/index.ts +++ b/packages/qa/src/index.ts @@ -3,6 +3,7 @@ export const toolName = 'release-qa'; export { parseCandidate, type Artifact, type Candidate } from './model/candidate.ts'; export { parseException, type Exception } from './model/exception.ts'; +export { parseLocalCandidate, type LocalArtifact, type LocalCandidate } from './model/local-candidate.ts'; export { parseProject, type EnvironmentProfile, type Project, type Suite } from './model/project.ts'; export { parseRequirement, type ExecutionMode, type Requirement, type RequirementKey } from './model/requirement.ts'; export { @@ -35,6 +36,8 @@ export { defaultProbes, inspectEnvironment, type EnvironmentProbes, type Inspect export { AssertionFailure, executeScenario, + type ArtifactRef, + type CandidateRef, type ExecutionContext, type Lifecycle, type Phase, diff --git a/packages/qa/src/model/local-candidate.ts b/packages/qa/src/model/local-candidate.ts new file mode 100644 index 0000000..aa37f0e --- /dev/null +++ b/packages/qa/src/model/local-candidate.ts @@ -0,0 +1,43 @@ +import { at, Collector, item, parseVersioned, type FieldSpec, type ParseResult } from './validate.ts'; + +/** One file to test for one environment profile, located relative to the manifest that lists it. */ +export interface LocalArtifact { + profile: string; + name: string; + /** Relative to the manifest's own directory, with `/` separators; it cannot leave that directory. */ + path: string; + sha256: string; +} + +/** + * A candidate built outside GitHub, for running a suite from a checkout (`release-qa run --candidate`). It names + * the files to test and their expected hashes; it carries none of the source or build provenance a GitHub + * candidate does, so it can never stand in for one at the merge gate. + */ +export interface LocalCandidate { + schemaVersion: 1; + id: string; + artifacts: LocalArtifact[]; +} + +const SPEC: FieldSpec = { required: ['id', 'artifacts'] }; +const ARTIFACT_SPEC: FieldSpec = { required: ['profile', 'name', 'path', 'sha256'] }; + +export function parseLocalCandidate(input: unknown): ParseResult { + return parseVersioned(input, SPEC, (c, rec) => { + const artifacts = (c.array(rec.artifacts, 'artifacts', { min: 1 }) ?? []).map((v, i) => readArtifact(c, v, item('artifacts', i))); + c.unique(artifacts.map((a, i) => ({ value: a?.profile, path: at(item('artifacts', i), 'profile') }))); + return { schemaVersion: 1, id: c.id(rec.id, 'id'), artifacts } as LocalCandidate; + }); +} + +function readArtifact(c: Collector, value: unknown, path: string): LocalArtifact | undefined { + const rec = c.record(value, path, ARTIFACT_SPEC); + if (rec === undefined) return undefined; + return { + profile: c.profileId(rec.profile, at(path, 'profile')), + name: c.fileName(rec.name, at(path, 'name')), + path: c.relativePath(rec.path, at(path, 'path')), + sha256: c.sha256(rec.sha256, at(path, 'sha256')), + } as LocalArtifact; +} diff --git a/packages/qa/src/runner/execute.ts b/packages/qa/src/runner/execute.ts index f21a95b..14c0a55 100644 --- a/packages/qa/src/runner/execute.ts +++ b/packages/qa/src/runner/execute.ts @@ -1,5 +1,4 @@ import type { ChildProcess, SpawnOptions } from 'node:child_process'; -import type { Candidate } from '../model/candidate.ts'; import type { EnvironmentProfile } from '../model/project.ts'; import type { Requirement, RequirementKey } from '../model/requirement.ts'; import type { MeasuredEnvironment, Outcome } from '../model/result.ts'; @@ -35,9 +34,24 @@ export interface ScenarioEvent { detail?: string; } +/** Which candidate a run tests. A full GitHub candidate record fits here; a local run needs only an identity. */ +export interface CandidateRef { + id: string; +} + +/** The candidate's file for this profile, already checked against its SHA-256 before anything ran. */ +export interface ArtifactRef { + name: string; + /** Absolute path to the verified file. */ + path: string; + sha256: string; +} + /** What a lifecycle hook or scenario may do. `own` and `spawn` record ownership before anything is used. */ export interface RunContext { - candidate: Candidate; + candidate: CandidateRef; + /** Present when the run tests a file; every hook and the steps see the same one. */ + artifact?: ArtifactRef; profile: EnvironmentProfile; testRoot: string; signal: AbortSignal; @@ -65,7 +79,8 @@ export interface Scenario { } export interface ExecutionContext { - candidate: Candidate; + candidate: CandidateRef; + artifact?: ArtifactRef; profile: EnvironmentProfile; testRoot: string; signal: AbortSignal; @@ -309,6 +324,7 @@ export async function executeScenario(context: ExecutionContext, scenario: Scena }; return { candidate: context.candidate, + ...(context.artifact === undefined ? {} : { artifact: context.artifact }), profile: context.profile, testRoot: root, signal, diff --git a/packages/qa/test/cli/args.test.ts b/packages/qa/test/cli/args.test.ts index 0defbbd..588aa9a 100644 --- a/packages/qa/test/cli/args.test.ts +++ b/packages/qa/test/cli/args.test.ts @@ -15,12 +15,12 @@ const failure = (argv: string[]): { error: string; json: boolean } => { describe('no command', () => { test('an empty argument list names the valid commands', () => { - expect(err([])).toMatch(/doctor.*designate.*status/); + expect(err([])).toMatch(/doctor.*designate.*status.*reset.*run.*resume/); }); test('an unrecognised first word names it and the valid commands', () => { expect(err(['fly'])).toMatch(/"fly"/); - expect(err(['fly'])).toMatch(/doctor.*designate.*status/); + expect(err(['fly'])).toMatch(/doctor.*designate.*status.*reset.*run.*resume/); }); test('json is reported even when there is no valid command at all', () => { @@ -107,3 +107,56 @@ describe('status', () => { expect(parseArgs(['status'])).toEqual({ ok: true, command: { name: 'status', root: undefined, json: false } }); }); }); + +describe('reset', () => { + test('--root is optional, like designate and status', () => { + expect(parseArgs(['reset'])).toEqual({ ok: true, command: { name: 'reset', root: undefined, json: false } }); + expect(parseArgs(['reset', '--root', 'r'])).toEqual({ ok: true, command: { name: 'reset', root: 'r', json: false } }); + }); +}); + +describe('run', () => { + const full = ['run', '--project', 'qa/project.json', '--candidate', 'candidate.json', '--profile', 'windows', '--suite', 'release']; + + test('reads the four required flags, and leaves --root and --state to the caller when absent', () => { + expect(parseArgs(full)).toEqual({ + ok: true, + command: { name: 'run', project: 'qa/project.json', candidate: 'candidate.json', profile: 'windows', suite: 'release', root: undefined, state: undefined, json: false }, + }); + }); + + test('reads --root, --state and --json when given', () => { + const result = parseArgs([...full, '--root', 'r', '--state', 's', '--json']); + expect(result).toMatchObject({ ok: true, command: { root: 'r', state: 's', json: true } }); + }); + + test.each([['--project'], ['--candidate'], ['--profile'], ['--suite']])('missing %s is an error naming it', (flag) => { + const index = full.indexOf(flag); + const argv = [...full.slice(0, index), ...full.slice(index + 2)]; + expect(err(argv)).toContain(flag); + }); +}); + +describe('resume', () => { + test('reads --run, and leaves --state to the caller', () => { + expect(parseArgs(['resume', '--run', 'run-1'])).toEqual({ ok: true, command: { name: 'resume', run: 'run-1', state: undefined, json: false } }); + }); + + test('--run is required', () => { + expect(err(['resume'])).toContain('--run'); + }); + + test('resume repeats what run was asked, so run\'s own flags are refused here', () => { + expect(err(['resume', '--run', 'run-1', '--profile', 'linux'])).toContain('profile'); + }); +}); + +describe('resume --run is a run id, never a path', () => { + test.each([['../../outside'], ['runs/run-1'], [String.raw`..\outside`], [String.raw`run\1`], ['.'], ['']])('%j is refused', (run) => { + expect(err(['resume', '--run', run])).toMatch(/run id/); + }); + + test('an ordinary run id is accepted', () => { + expect(parseArgs(['resume', '--run', 'run-20260923T101500Z-a1b2c3']).ok).toBe(true); + }); +}); diff --git a/packages/qa/test/cli/candidate.test.ts b/packages/qa/test/cli/candidate.test.ts new file mode 100644 index 0000000..c8193e6 --- /dev/null +++ b/packages/qa/test/cli/candidate.test.ts @@ -0,0 +1,171 @@ +import { createHash } from 'node:crypto'; +import { mkdir, mkdtemp, realpath, rm, symlink, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { loadCandidate } from '../../src/cli/candidate.ts'; + +const dirs: string[] = []; +afterEach(async () => { + await Promise.allSettled(dirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +const sha = (bytes: string): string => createHash('sha256').update(bytes).digest('hex'); + +async function manifestWith(files: Record, artifacts: unknown[], id = 'local-1'): Promise<{ dir: string; path: string }> { + const dir = await mkdtemp(join(tmpdir(), 'qa-cli-candidate-')); + dirs.push(dir); + for (const [name, bytes] of Object.entries(files)) { + await mkdir(join(dir, name, '..'), { recursive: true }); + await writeFile(join(dir, name), bytes); + } + const path = join(dir, 'candidate.json'); + await writeFile(path, JSON.stringify({ schemaVersion: 1, id, artifacts })); + return { dir, path }; +} + +const loaded = async (...args: Parameters) => { + const result = await loadCandidate(...args); + if (!result.ok) throw new Error(result.error); + return result; +}; +const failed = async (...args: Parameters): Promise => { + const result = await loadCandidate(...args); + if (result.ok) throw new Error('expected loading to fail'); + return result.error; +}; + +describe('loading a candidate for one profile', () => { + test('picks the artifact for the profile, resolves it next to the manifest, and verifies its bytes', async () => { + const { dir, path } = await manifestWith({ 'dist/setup.exe': 'installer bytes' }, [ + { profile: 'windows', name: 'setup.exe', path: 'dist/setup.exe', sha256: sha('installer bytes') }, + { profile: 'linux', name: 'app.deb', path: 'dist/app.deb', sha256: 'c'.repeat(64) }, + ]); + + const result = await loaded(path, 'windows'); + + expect(result.candidate).toEqual({ id: 'local-1' }); + // Hooks get the file's real location, so nothing on the way can be swapped to point elsewhere afterwards. + expect(result.artifact).toEqual({ name: 'setup.exe', path: await realpath(join(dir, 'dist', 'setup.exe')), sha256: sha('installer bytes') }); + }); + + test('a file whose bytes do not match the manifest is refused, naming both hashes', async () => { + const { path } = await manifestWith({ 'setup.exe': 'tampered' }, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('original') }]); + const error = await failed(path, 'windows'); + expect(error).toContain(sha('original')); + expect(error).toContain(sha('tampered')); + }); + + test('another profile\'s artifact is never hashed or used', async () => { + // The linux file does not exist; loading for windows must not touch it. + const { path } = await manifestWith({ 'setup.exe': 'x' }, [ + { profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }, + { profile: 'linux', name: 'app.deb', path: 'missing/app.deb', sha256: 'c'.repeat(64) }, + ]); + expect((await loadCandidate(path, 'windows')).ok).toBe(true); + }); + + test('a profile the manifest has no artifact for is refused, naming the profiles it does have', async () => { + const { path } = await manifestWith({ 'setup.exe': 'x' }, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }]); + const error = await failed(path, 'linux'); + expect(error).toContain('linux'); + expect(error).toContain('windows'); + }); + + test('a missing artifact file is refused', async () => { + const { path } = await manifestWith({}, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }]); + expect(await failed(path, 'windows')).toContain('setup.exe'); + }); + + test('a directory where the artifact should be is refused', async () => { + const { dir, path } = await manifestWith({}, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }]); + await mkdir(join(dir, 'setup.exe')); + expect(await failed(path, 'windows')).toMatch(/not a file/); + }); + + test.each([ + ['a missing manifest', null], + ['a manifest that is not JSON', 'not json'], + ['a manifest that fails validation', JSON.stringify({ schemaVersion: 1, id: 'x', artifacts: [] })], + ])('%s is refused, not thrown', async (_label, content) => { + const dir = await mkdtemp(join(tmpdir(), 'qa-cli-candidate-')); + dirs.push(dir); + const path = join(dir, 'candidate.json'); + if (content !== null) await writeFile(path, content); + expect((await loadCandidate(path, 'windows')).ok).toBe(false); + }); +}); + +describe('the artifact path cannot leave the manifest directory through a link', () => { + test('a directory link pointing outside is refused, and the file behind it is not treated as the candidate', async () => { + const outside = await mkdtemp(join(tmpdir(), 'qa-cli-outside-')); + dirs.push(outside); + await writeFile(join(outside, 'setup.exe'), 'somebody else\'s file'); + const { dir, path } = await manifestWith({}, [{ profile: 'windows', name: 'setup.exe', path: 'dist/setup.exe', sha256: sha('somebody else\'s file') }]); + await symlink(outside, join(dir, 'dist'), 'junction'); + + const error = await failed(path, 'windows'); + + expect(error).toMatch(/outside/); + }); +}); + +describe('the path hooks are given', () => { + test('a link inside the manifest directory is resolved, so hooks get a path with no link left to swap', async () => { + const { dir, path } = await manifestWith({ 'real-dist/setup.exe': 'x' }, [{ profile: 'windows', name: 'setup.exe', path: 'dist/setup.exe', sha256: sha('x') }]); + await symlink(join(dir, 'real-dist'), join(dir, 'dist'), 'junction'); + const result = await loaded(path, 'windows'); + expect(result.artifact.path).toBe(await realpath(join(dir, 'real-dist', 'setup.exe'))); + }); +}); + +describe('a real path that cannot be resolved', () => { + afterEach(() => { + vi.doUnmock('node:fs/promises'); + vi.resetModules(); + }); + + test('is a refusal, not a thrown error', async () => { + const { path } = await manifestWith({ 'setup.exe': 'x' }, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }]); + vi.doMock('node:fs/promises', async (importOriginal) => ({ ...(await importOriginal()), realpath: async () => { throw new Error('EACCES: resolving denied'); } })); + vi.resetModules(); + const { loadCandidate: loadWithBrokenRealpath } = await import('../../src/cli/candidate.ts'); + const result = await loadWithBrokenRealpath(path, 'windows'); + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toContain('resolving denied'); + }); +}); + +describe('a link swapped while the artifact is being hashed', () => { + afterEach(() => { + vi.doUnmock('node:fs/promises'); + vi.resetModules(); + }); + + test('is refused: the check before hashing would describe some other file', async () => { + const { dir, path } = await manifestWith({ 'setup.exe': 'x' }, [{ profile: 'windows', name: 'setup.exe', path: 'setup.exe', sha256: sha('x') }]); + const file = join(dir, 'setup.exe'); + let resolvedFile = 0; + // The first resolution of the artifact sees the real file; the one after hashing sees it moved elsewhere. + vi.doMock('node:fs/promises', async (importOriginal) => { + const real = await importOriginal(); + return { + ...real, + realpath: async (target: string) => { + const resolved = await real.realpath(target); + if (target !== file) return resolved; + resolvedFile += 1; + return resolvedFile === 1 ? resolved : `${resolved}.swapped`; + }, + }; + }); + vi.resetModules(); + const { loadCandidate: loadWithSwap } = await import('../../src/cli/candidate.ts'); + + const result = await loadWithSwap(path, 'windows'); + + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toContain('changed location'); + expect(resolvedFile).toBe(2); + }); +}); diff --git a/packages/qa/test/cli/consumer.test.ts b/packages/qa/test/cli/consumer.test.ts new file mode 100644 index 0000000..73e6328 --- /dev/null +++ b/packages/qa/test/cli/consumer.test.ts @@ -0,0 +1,92 @@ +import { writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { afterEach, describe, expect, test } from 'vitest'; +import { loadConsumer } from '../../src/cli/consumer.ts'; +import { loadProject } from '../../src/cli/project.ts'; +import { selectPlan } from '../../src/cli/plan.ts'; +import { writeConsumer } from '../fixtures/consumer.ts'; +import { cleanUpProcessesAndRoots } from '../fixtures/processes.ts'; + +afterEach(cleanUpProcessesAndRoots); + +async function planFor(projectPath: string, profile: string) { + const loaded = await loadProject(projectPath); + if (!loaded.ok) throw new Error(loaded.error); + const plan = selectPlan(loaded.project, profile, 'release'); + if (!plan.ok) throw new Error(plan.error); + return { project: loaded.project, plan }; +} + +const failure = async (...args: Parameters): Promise => { + const result = await loadConsumer(...args); + if (result.ok) throw new Error('expected loading to fail'); + return result.error; +}; + +describe('loading a consumer project\'s code', () => { + test('binds each automated requirement to the scenario with the same id, in plan order', async () => { + const consumer = await writeConsumer({ scenarios: { startup: 'pass', persistence: 'pass' } }); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + + const result = await loadConsumer(consumer.projectPath, project, plan.automated); + + if (!result.ok) throw new Error(result.error); + expect(result.scenarios.map((s) => [s.id, s.requirement.key])).toEqual([ + ['startup', `${consumer.profile}/startup`], + ['persistence', `${consumer.profile}/persistence`], + ]); + expect(typeof result.lifecycle.install).toBe('function'); + }); + + test('a requirement no scenario file defines is refused before anything runs, naming it', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + const missing = { ...plan.automated[0]!, key: `${consumer.profile}/uninstall` as const }; + expect(await failure(consumer.projectPath, project, [...plan.automated, missing])).toContain('uninstall'); + }); + + test('a lifecycle module without a complete lifecycle export is refused, naming what is missing', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'lifecycle.ts'), 'export const lifecycle = { install: async () => {} };'); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + expect(await failure(consumer.projectPath, project, plan.automated)).toMatch(/reset/); + }); + + test('a scenario file without a scenarios array is refused', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'scenarios.ts'), 'export const something = 1;'); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + expect(await failure(consumer.projectPath, project, plan.automated)).toMatch(/scenarios/); + }); + + test('two scenario files defining the same id are refused: which one runs would be ambiguous', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'again.ts'), "export const scenarios = [{ id: 'persistence', steps: async () => {} }];"); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + const withTwoFiles = { ...project, scenarioFiles: ['scenarios.ts', 'again.ts'] }; + expect(await failure(consumer.projectPath, withTwoFiles, plan.automated)).toMatch(/persistence.*more than once/); + }); + + test('a module that throws while loading is a refusal with its message, not a crash', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'lifecycle.ts'), "throw new Error('cannot find the installer tool');"); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + expect(await failure(consumer.projectPath, project, plan.automated)).toContain('cannot find the installer tool'); + }); +}); + +describe('module shapes that throw while being inspected', () => { + test('a lifecycle whose hook is a throwing getter is a refusal, not a crash', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'lifecycle.ts'), "export const lifecycle = { get install() { throw new Error('getter exploded'); } };"); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + expect(await failure(consumer.projectPath, project, plan.automated)).toContain('getter exploded'); + }); + + test('a scenario definition whose id is a throwing getter is a refusal, not a crash', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await writeFile(join(consumer.dir, 'qa', 'scenarios.ts'), "export const scenarios = [{ get id() { throw new Error('id exploded'); }, steps: async () => {} }];"); + const { project, plan } = await planFor(consumer.projectPath, consumer.profile); + expect(await failure(consumer.projectPath, project, plan.automated)).toContain('id exploded'); + }); +}); diff --git a/packages/qa/test/cli/environment-commands.test.ts b/packages/qa/test/cli/environment-commands.test.ts index 56a9d01..6d4ef0f 100644 --- a/packages/qa/test/cli/environment-commands.test.ts +++ b/packages/qa/test/cli/environment-commands.test.ts @@ -1,10 +1,12 @@ -import { mkdir, realpath, writeFile } from 'node:fs/promises'; +import { mkdir, realpath, stat, writeFile } from 'node:fs/promises'; import { join } from 'node:path'; import { afterEach, describe, expect, test, vi } from 'vitest'; -import { runDesignate, runStatus } from '../../src/cli/environment-commands.ts'; -import { spawnOwned, markDirty } from '../../src/runner/resources.ts'; +import { runDesignate, runReset, runStatus } from '../../src/cli/environment-commands.ts'; +import { acquireTestRoot, markDirty, readDirty, readLedger, recordOwned, spawnOwned } from '../../src/runner/resources.ts'; import { cleanUpProcessesAndRoots, makeTempDir, makeTestRoot, trackProcess } from '../fixtures/processes.ts'; +// Reading a process's identity starts PowerShell on Windows, which can take seconds on a busy CI runner. +vi.setConfig({ testTimeout: 30_000 }); afterEach(cleanUpProcessesAndRoots); describe('designate', () => { @@ -72,6 +74,55 @@ describe('status', () => { }); }); +describe('reset', () => { + test('reaps what an earlier run left owned and clears the dirty marker', async () => { + const root = await makeTestRoot(); + const leftover = join(root, 'installed-app'); + await mkdir(leftover); + await recordOwned(root, { kind: 'path', path: leftover, label: 'installed app' }); + await markDirty(root, 'a run crashed'); + + const result = await runReset(root); + + expect(result).toEqual({ ok: true, root: await realpath(root), failures: [] }); + expect(await readDirty(root)).toBeUndefined(); + expect(await readLedger(root)).toEqual([]); + expect(await stat(leftover).then(() => true, () => false)).toBe(false); + }); + + test('reports what it could not clean, and leaves the environment dirty', async () => { + const root = await makeTestRoot(); + const outside = await makeTempDir('qa-outside-'); + await recordOwned(root, { kind: 'path', path: outside, label: 'not really ours' }); + await markDirty(root, 'a run crashed'); + + const result = await runReset(root); + + expect(result.ok && result.failures.join(' ')).toMatch(/not really ours.*outside-test-root/); + expect(await readDirty(root)).toBeDefined(); + expect(await stat(outside).then(() => true, () => false)).toBe(true); + }); + + test('refuses a directory that is not a designated test root, touching nothing', async () => { + const root = await makeTestRoot(false); + const result = await runReset(root); + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toContain('missing-marker'); + }); + + test('refuses while a run holds the root, so it cannot reap resources out from under it', async () => { + const root = await makeTestRoot(); + const lock = await acquireTestRoot(root); + try { + const result = await runReset(root); + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toContain(String(process.pid)); + } finally { + if (lock.ok) await lock.release(); + } + }); +}); + describe('the root check itself failing', () => { // Torn down here rather than at the end of the test, so a failed assertion cannot leave the mock registered and // confusingly break whatever test happens to run after this one. diff --git a/packages/qa/test/cli/executable.test.ts b/packages/qa/test/cli/executable.test.ts index e3f3d5c..c8fc05f 100644 --- a/packages/qa/test/cli/executable.test.ts +++ b/packages/qa/test/cli/executable.test.ts @@ -1,16 +1,22 @@ // Proves the CLI genuinely runs as `node packages/qa/src/cli/main.ts ...`, with no build step, from another process // (not just imported as a module inside the test worker) — the thing test/no-build.test.ts proves for the package -// as a whole, exercised here specifically through the paths the plan calls out: malformed args and an unsupported -// profile. "Missing candidate" and "cancellation" apply to `run`, which this build does not implement yet. -import { execFile } from 'node:child_process'; -import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +// as a whole, exercised here through the paths the plan calls out: malformed args, an unsupported profile, a missing +// candidate and cancellation, plus a genuine crash in the middle of a scenario followed by `resume`. +import { execFile, spawn } from 'node:child_process'; +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { dirname, join } from 'node:path'; import { tmpdir } from 'node:os'; import { fileURLToPath } from 'node:url'; import { promisify } from 'node:util'; -import { afterEach, describe, expect, test } from 'vitest'; +import { afterEach, describe, expect, test, vi } from 'vitest'; import { EXIT } from '../../src/cli/main.ts'; -import { hostOs } from '../fixtures/processes.ts'; +import { readRun } from '../../src/runner/journal.ts'; +import { writeConsumer, type Consumer } from '../fixtures/consumer.ts'; +import { cleanUpProcessesAndRoots, eventually, hostOs, trackProcess } from '../fixtures/processes.ts'; + +// Each case starts a Node process, and reading a process's identity starts PowerShell on Windows. +vi.setConfig({ testTimeout: 60_000 }); +afterEach(cleanUpProcessesAndRoots); const run = promisify(execFile); const cliPath = join(dirname(fileURLToPath(import.meta.url)), '..', '..', 'src', 'cli', 'main.ts'); @@ -21,8 +27,12 @@ afterEach(async () => { }); async function run_(...args: string[]): Promise<{ stdout: string; stderr: string; code: number }> { + return runIn(process.cwd(), ...args); +} + +async function runIn(cwd: string, ...args: string[]): Promise<{ stdout: string; stderr: string; code: number }> { try { - const { stdout, stderr } = await run(process.execPath, [cliPath, ...args]); + const { stdout, stderr } = await run(process.execPath, [cliPath, ...args], { cwd }); return { stdout, stderr, code: 0 }; } catch (error) { const failure = error as { stdout: string; stderr: string; code: number }; @@ -98,3 +108,59 @@ describe('the CLI executable', () => { expect(printed).toMatchObject({ ok: true, profile: 'here' }); }); }); + +describe('running a suite through the executable', () => { + const runArgs = (consumer: Consumer): string[] => ['run', '--project', consumer.projectPath, '--candidate', consumer.candidatePath, '--profile', consumer.profile, '--suite', 'release']; + + test('a missing candidate manifest exits 3 before anything runs', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + expect((await runIn(consumer.dir, 'designate')).code).toBe(EXIT.ok); + await rm(consumer.candidatePath); + const result = await runIn(consumer.dir, ...runArgs(consumer)); + expect(result.code).toBe(EXIT.infrastructure); + expect(result.stderr).toContain('candidate.json'); + expect(await readFile(consumer.logPath, 'utf8').catch(() => '')).toBe(''); + }); + + test('a process that dies in the middle of a scenario leaves a run that resume completes, with the crash on record', async () => { + const consumer = await writeConsumer({ scenarios: { startup: 'pass', persistence: 'pass' }, crashOnce: 'persistence' }); + expect((await runIn(consumer.dir, 'designate')).code).toBe(EXIT.ok); + + const crashed = await runIn(consumer.dir, ...runArgs(consumer)); + expect(crashed.code).toBe(70); // the scenario itself killed the process + const runId = /run (\S+) started/.exec(crashed.stderr)?.[1]; + expect(runId).toBeDefined(); + + const resumed = await runIn(consumer.dir, 'resume', '--run', runId as string, '--json'); + + expect(resumed.code).toBe(EXIT.ok); + const summary = JSON.parse(resumed.stdout) as { results: Array<{ requirement: string; outcome: string; carried?: boolean }> }; + expect(summary.results.map((r) => [r.requirement.split('/')[1], r.outcome, r.carried ?? false])).toEqual([ + ['startup', 'passed', true], + ['persistence', 'passed', false], + ]); + const attempts = (await readRun(join(consumer.dir, '.release-qa', 'runs', runId as string))).attempts.filter((a) => a.requirement.endsWith('/persistence')); + expect(attempts.map((a) => a.outcome)).toEqual(['interrupted', 'passed']); + expect(attempts[1]?.retryOf).toBe(attempts[0]?.id); + }); + + // Node on Windows cannot deliver SIGINT to another process (kill() terminates it outright); a console Ctrl+C + // does reach the handler there, but cannot be sent from a test. Linux CI exercises the real signal path. + test.skipIf(process.platform === 'win32')('SIGINT cancels the running scenario, cleans up, and exits 3 with a summary', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'hang', uninstall: 'pass' } }); + expect((await runIn(consumer.dir, 'designate')).code).toBe(EXIT.ok); + const child = trackProcess(spawn(process.execPath, [cliPath, ...runArgs(consumer), '--json'], { cwd: consumer.dir, stdio: ['ignore', 'pipe', 'pipe'] })); + let stdout = ''; + child.stdout.on('data', (chunk: Buffer) => { stdout += chunk.toString(); }); + // 'close', not 'exit': only then are the output streams flushed, so the summary is complete. + const exited = new Promise((resolveExit) => child.on('close', (code) => resolveExit(code))); + + await eventually(async () => (await readFile(consumer.logPath, 'utf8').catch(() => '')).includes('steps:persistence'), 30_000); + child.kill('SIGINT'); + + expect(await exited).toBe(EXIT.infrastructure); + const summary = JSON.parse(stdout) as { results: Array<{ outcome: string }> }; + expect(summary.results.map((r) => r.outcome)).toEqual(['cancelled', 'not-run']); + expect((await readFile(consumer.logPath, 'utf8')).trim().split('\n').at(-1)).toMatch(/^cleanup /); + }); +}); diff --git a/packages/qa/test/cli/main.test.ts b/packages/qa/test/cli/main.test.ts index 56a347c..0059214 100644 --- a/packages/qa/test/cli/main.test.ts +++ b/packages/qa/test/cli/main.test.ts @@ -1,9 +1,14 @@ -import { mkdir, realpath, rm, writeFile } from 'node:fs/promises'; +import { mkdir, readFile, realpath, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { afterEach, describe, expect, test } from 'vitest'; -import { EXIT, main } from '../../src/cli/main.ts'; -import { hostOs, hostProfile } from '../fixtures/processes.ts'; +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { EXIT, main, resumeHint } from '../../src/cli/main.ts'; +import { writeConsumer } from '../fixtures/consumer.ts'; +import { cleanUpProcessesAndRoots, eventually, hostOs, hostProfile } from '../fixtures/processes.ts'; + +// Reading a process's identity starts PowerShell on Windows, which can take seconds on a busy CI runner. +vi.setConfig({ testTimeout: 30_000 }); +afterEach(cleanUpProcessesAndRoots); const dirs: string[] = []; afterEach(async () => { @@ -149,3 +154,124 @@ describe('designate and status', () => { expect(out.log.join(' ')).toContain('designated: false'); }); }); + +describe('run, resume and reset', () => { + async function designated(consumerDir: string): Promise { + const out = io(); + expect(await main(['designate'], out.sink, () => consumerDir)).toBe(EXIT.ok); + } + const runArgs = (consumer: Awaited>) => ['run', '--project', consumer.projectPath, '--candidate', consumer.candidatePath, '--profile', consumer.profile, '--suite', 'release']; + + test('a passing run exits 0 and prints its summary; root and state default to .release-qa under the directory', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await designated(consumer.dir); + const out = io(); + + const code = await main([...runArgs(consumer), '--json'], out.sink, () => consumer.dir); + + expect(code).toBe(EXIT.ok); + const summary = JSON.parse(out.log.join('')) as { runId: string; results: Array<{ outcome: string }> }; + expect(summary.results.map((r) => r.outcome)).toEqual(['passed']); + expect(await realpath(join(consumer.dir, '.release-qa', 'runs', summary.runId))).toBeTruthy(); + }); + + test('the run id is announced on stderr as soon as the run exists, so it can be resumed even if the process dies', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await designated(consumer.dir); + const order: string[] = []; + const sink = { log: (line: string) => order.push(`out:${line}`), error: (line: string) => order.push(`err:${line}`) }; + await main([...runArgs(consumer), '--json'], sink, () => consumer.dir); + const { runId } = JSON.parse(order.find((l) => l.startsWith('out:'))!.slice(4)) as { runId: string }; + const announcement = order.findIndex((l) => l.startsWith('err:') && l.includes(runId)); + expect(announcement).toBeGreaterThanOrEqual(0); + expect(announcement).toBeLessThan(order.findIndex((l) => l.startsWith('out:'))); + }); + + test('the resume hint names a custom --state, so copying it after a crash finds the run', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await designated(consumer.dir); + const out = io(); + await main([...runArgs(consumer), '--state', 'elsewhere'], out.sink, () => consumer.dir); + const hint = out.error.find((line) => line.includes('resume --run')) ?? ''; + expect(hint).toContain('--state'); + expect(hint).toContain(join(consumer.dir, 'elsewhere')); + }); + + test('a failing scenario makes the run exit 1', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'fail' } }); + await designated(consumer.dir); + expect(await main(runArgs(consumer), io().sink, () => consumer.dir)).toBe(EXIT.scenarioFailure); + }); + + test('a manual requirement left over makes the run exit 2', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' }, manual: ['audio'] }); + await designated(consumer.dir); + expect(await main(runArgs(consumer), io().sink, () => consumer.dir)).toBe(EXIT.missingPrerequisite); + }); + + test('a candidate that does not verify exits 3 before anything runs', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + await designated(consumer.dir); + await writeFile(join(consumer.dir, 'setup.bin'), 'tampered'); + const out = io(); + expect(await main([...runArgs(consumer), '--json'], out.sink, () => consumer.dir)).toBe(EXIT.infrastructure); + expect((JSON.parse(out.error.join('')) as { error: string }).error).toContain('does not match'); + }); + + test('cancelling through the signal exits 3, and resume then finishes the run', async () => { + const consumer = await writeConsumer({ scenarios: { startup: 'pass', persistence: 'hang' } }); + await designated(consumer.dir); + const controller = new AbortController(); + const out = io(); + const running = main([...runArgs(consumer), '--json'], out.sink, () => consumer.dir, controller.signal); + try { + await eventually(async () => (await readFile(consumer.logPath, 'utf8').catch(() => '')).includes('steps:persistence')); + } finally { + // Even if the wait above fails, cancel the run and let it finish, so the failure is that assertion, not a timeout. + controller.abort(); + } + expect(await running).toBe(EXIT.infrastructure); + const { runId } = JSON.parse(out.log.join('')) as { runId: string }; + + await rm(consumer.holdPath); + const resumed = io(); + expect(await main(['resume', '--run', runId, '--json'], resumed.sink, () => consumer.dir)).toBe(EXIT.ok); + const summary = JSON.parse(resumed.log.join('')) as { results: Array<{ outcome: string; carried?: boolean }> }; + expect(summary.results.map((r) => [r.outcome, r.carried ?? false])).toEqual([['passed', true], ['passed', false]]); + }); + + test('resuming a run that does not exist exits 3', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + expect(await main(['resume', '--run', 'run-nope'], io().sink, () => consumer.dir)).toBe(EXIT.infrastructure); + }); + + test('reset on a clean designated root exits 0; on an undesignated one, 3', async () => { + const consumer = await writeConsumer({ scenarios: { persistence: 'pass' } }); + expect(await main(['reset'], io().sink, () => consumer.dir)).toBe(EXIT.infrastructure); + await designated(consumer.dir); + expect(await main(['reset'], io().sink, () => consumer.dir)).toBe(EXIT.ok); + }); +}); + +describe('the resume hint', () => { + test('a plain state directory is written as it is', () => { + expect(resumeHint('run-1', String.raw`C:\qa\runs`)).toBe(String.raw`resume --run run-1 --state C:\qa\runs`); + expect(resumeHint('run-1', '/srv/qa/runs')).toBe('resume --run run-1 --state /srv/qa/runs'); + }); + + test('without a custom state directory there is nothing to add', () => { + expect(resumeHint('run-1', undefined)).toBe('resume --run run-1'); + }); + + // Single quotes are literal in both bash and PowerShell, so nothing inside them is expanded or substituted. + test.each([[String.raw`C:\My QA\runs`], ['/tmp/$HOME/runs'], ['/tmp/`id`/runs'], ['/tmp/a"b/runs']])('%s is single-quoted so it pastes safely', (dir) => { + expect(resumeHint('run-1', dir)).toBe(`resume --run run-1 --state '${dir}'`); + }); + + test('a path containing a single quote is not put into a command at all, but still named', () => { + const hint = resumeHint('run-1', "/tmp/it's/runs"); + expect(hint).not.toMatch(/--state '/); + expect(hint).toContain("/tmp/it's/runs"); + expect(hint).toContain('resume --run run-1'); + }); +}); diff --git a/packages/qa/test/cli/plan.test.ts b/packages/qa/test/cli/plan.test.ts new file mode 100644 index 0000000..16b0b6d --- /dev/null +++ b/packages/qa/test/cli/plan.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, test } from 'vitest'; +import { selectPlan } from '../../src/cli/plan.ts'; +import type { Project } from '../../src/model/project.ts'; +import type { Requirement } from '../../src/model/requirement.ts'; + +const req = (key: `${string}/${string}`, mode: Requirement['mode'] = 'automated'): Requirement => ({ key, mode, title: key, capabilities: [] }); + +const project = (): Project => ({ + schemaVersion: 1, + projectId: 'sample', + releaseBranch: 'main', + profiles: [ + { id: 'windows', os: 'windows', arch: 'x86_64' }, + { id: 'linux', os: 'linux', arch: 'x86_64' }, + ], + requirements: [req('windows/persistence'), req('windows/audio', 'manual'), req('linux/persistence'), req('windows/startup')], + suites: [ + { id: 'release', requirements: ['windows/persistence', 'windows/audio', 'linux/persistence'] }, + { id: 'linux-only', requirements: ['linux/persistence'] }, + ], + scenarioFiles: [], + lifecycleModule: 'lifecycle.ts', + workflows: { prepare: 'a.yml', gate: 'b.yml', publish: 'c.yml' }, + markers: { releaseNotes: 'release-notes', qa: 'qa' }, +}); + +describe('selecting what a run covers', () => { + test('takes the suite\'s requirements for this profile only, split into automated and manual, in suite order', () => { + const plan = selectPlan(project(), 'windows', 'release'); + expect(plan).toEqual({ ok: true, profile: project().profiles[0], automated: [req('windows/persistence')], manual: [req('windows/audio', 'manual')] }); + }); + + test('requirements outside the suite are not included, even for the same profile', () => { + const plan = selectPlan(project(), 'windows', 'release'); + expect(plan.ok && plan.automated.map((r) => r.key)).not.toContain('windows/startup'); + }); + + test.each([ + ['an unknown profile', 'macos', 'release', /profile "macos".*windows, linux/], + ['an unknown suite', 'windows', 'nightly', /suite "nightly".*release, linux-only/], + ['a suite with nothing for this profile', 'windows', 'linux-only', /nothing for profile "windows"/], + ])('%s is refused with a message saying what exists', (_label, profile, suite, pattern) => { + const plan = selectPlan(project(), profile, suite); + expect(plan.ok).toBe(false); + expect(!plan.ok && plan.error).toMatch(pattern); + }); +}); diff --git a/packages/qa/test/cli/run.test.ts b/packages/qa/test/cli/run.test.ts new file mode 100644 index 0000000..cb3bb0d --- /dev/null +++ b/packages/qa/test/cli/run.test.ts @@ -0,0 +1,308 @@ +import { readFile, realpath, rm, stat, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { exitCodeOf, resumeRun, startRun, type RequirementResult, type RunOptions, type RunSummary } from '../../src/cli/run.ts'; +import { readRun } from '../../src/runner/journal.ts'; +import { writeConsumer, type Behaviour, type Consumer } from '../fixtures/consumer.ts'; +import { cleanUpProcessesAndRoots, makeTempDir, makeTestRoot } from '../fixtures/processes.ts'; + +// Reading a process's identity starts PowerShell on Windows, which can take seconds on a busy CI runner. +vi.setConfig({ testTimeout: 30_000 }); +afterEach(cleanUpProcessesAndRoots); + +const exists = (path: string) => stat(path).then(() => true, () => false); +const calls = async (consumer: Consumer): Promise => (await readFile(consumer.logPath, 'utf8').catch(() => '')).split('\n').filter(Boolean); + +async function setUp(scenarios: Record, extra: { manual?: string[] } = {}) { + const consumer = await writeConsumer({ scenarios, ...extra }); + const root = await makeTestRoot(); + const stateDir = join(await makeTempDir('qa-state-'), 'runs'); + const controller = new AbortController(); + const options: RunOptions = { + stateDir, + signal: controller.signal, + probes: { display: async () => true, audio: async () => true }, + }; + const invocation = { project: consumer.projectPath, candidate: consumer.candidatePath, profile: consumer.profile, suite: 'release', root }; + return { consumer, root, stateDir, controller, options, invocation }; +} + +async function started(...args: Parameters): Promise { + const result = await startRun(...args); + if (!result.ok) throw new Error(result.error); + return result.summary; +} +async function resumed(...args: Parameters): Promise { + const result = await resumeRun(...args); + if (!result.ok) throw new Error(result.error); + return result.summary; +} +const outcomes = (summary: RunSummary) => Object.fromEntries(summary.results.map((r) => [r.requirement.split('/')[1], r.outcome])); + +describe('starting a run', () => { + test('runs every automated scenario, hands each hook the verified artifact, and records it all in the journal', async () => { + const { consumer, stateDir, options, invocation } = await setUp({ startup: 'pass', persistence: 'pass' }); + + const summary = await started(invocation, options); + + expect(summary.exitCode).toBe(0); + expect(outcomes(summary)).toEqual({ startup: 'passed', persistence: 'passed' }); + // Hooks get the file's real path (on some machines not byte-identical to the one it was written through). + const artifact = await realpath(join(consumer.dir, 'setup.bin')); + expect(await calls(consumer)).toEqual([ + `install ${artifact}`, `reset ${artifact}`, `launch ${artifact}`, `steps:startup ${artifact}`, `cleanup ${artifact}`, + `install ${artifact}`, `reset ${artifact}`, `launch ${artifact}`, `steps:persistence ${artifact}`, `cleanup ${artifact}`, + ]); + + const state = await readRun(join(stateDir, summary.runId)); + expect(state.events.map((e) => e.type)).toEqual(['run-started', 'checkpoint', 'attempt-recorded', 'checkpoint', 'attempt-recorded']); + expect(state.events[0]).toMatchObject({ data: { runId: summary.runId, candidateId: 'local-1', profile: consumer.profile } }); + expect(state.attempts.map((a) => [a.requirement, a.outcome])).toEqual([ + [`${consumer.profile}/startup`, 'passed'], + [`${consumer.profile}/persistence`, 'passed'], + ]); + expect(await exists(join(stateDir, summary.runId, 'summary.json'))).toBe(true); + }); + + test('the machine id is a generated token, not the host name, and stays the same across runs', async () => { + const { stateDir, options, invocation } = await setUp({ persistence: 'pass' }); + const first = await started(invocation, options); + const second = await started(invocation, options); + const machineOf = async (runId: string) => (await readRun(join(stateDir, runId))).events[0]?.data as { machineId: string }; + const machine = (await machineOf(first.runId)).machineId; + expect(machine).toBe((await machineOf(second.runId)).machineId); + expect(machine).not.toContain((await import('node:os')).hostname()); + }); + + test('one scenario failing its assertion does not stop the others, and the run exits 1', async () => { + const { options, invocation } = await setUp({ startup: 'fail', persistence: 'pass' }); + const summary = await started(invocation, options); + expect(outcomes(summary)).toEqual({ startup: 'failed', persistence: 'passed' }); + expect(summary.exitCode).toBe(1); + }); + + test('a scenario that breaks without an assertion is interrupted, and the run exits 3', async () => { + const { options, invocation } = await setUp({ persistence: 'throw' }); + const summary = await started(invocation, options); + expect(outcomes(summary)).toEqual({ persistence: 'interrupted' }); + expect(summary.exitCode).toBe(3); + }); + + test('manual requirements are listed as work left, never run, and the run exits 2', async () => { + const { consumer, stateDir, options, invocation } = await setUp({ persistence: 'pass' }, { manual: ['audio'] }); + const summary = await started(invocation, options); + expect(outcomes(summary)).toEqual({ persistence: 'passed', audio: 'manual' }); + expect(summary.exitCode).toBe(2); + expect((await readRun(join(stateDir, summary.runId))).attempts.map((a) => a.requirement)).toEqual([`${consumer.profile}/persistence`]); + }); + + test('a missing prerequisite blocks the scenario without installing anything, and the run exits 2', async () => { + const { consumer, options, invocation } = await setUp({ persistence: 'pass' }); + const undesignated = await makeTempDir('qa-undesignated-'); + const summary = await started({ ...invocation, root: undesignated }, options); + expect(outcomes(summary)).toEqual({ persistence: 'blocked' }); + expect(summary.exitCode).toBe(2); + expect(await calls(consumer)).toEqual([]); + }); + + test('a candidate whose bytes changed is refused before anything is installed or recorded', async () => { + const { consumer, stateDir, options, invocation } = await setUp({ persistence: 'pass' }); + await writeFile(join(consumer.dir, 'setup.bin'), 'tampered'); + const result = await startRun(invocation, options); + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toContain('does not match'); + expect(await calls(consumer)).toEqual([]); + expect(await exists(stateDir)).toBe(false); + }); + + test('an unknown suite or profile is refused before anything is recorded', async () => { + const { stateDir, options, invocation } = await setUp({ persistence: 'pass' }); + expect((await startRun({ ...invocation, suite: 'nightly' }, options)).ok).toBe(false); + expect((await startRun({ ...invocation, profile: 'plan9' }, options)).ok).toBe(false); + expect(await exists(stateDir)).toBe(false); + }); + + test('cancellation stops the running scenario, still cleans up, and starts nothing further; the run exits 3', async () => { + const { consumer, controller, options, invocation } = await setUp({ startup: 'pass', persistence: 'hang', uninstall: 'pass' }); + const summary = await started(invocation, { + ...options, + onEvent: (event) => { + if (event.scenario === 'persistence' && event.phase === 'steps' && event.status === 'started') controller.abort(); + }, + }); + expect(outcomes(summary)).toEqual({ startup: 'passed', persistence: 'cancelled', uninstall: 'not-run' }); + expect(summary.exitCode).toBe(3); + const log = await calls(consumer); + expect(log.at(-1)).toMatch(/^cleanup /); + expect(log.some((line) => line.startsWith('steps:uninstall'))).toBe(false); + }); +}); + +describe('resuming a run', () => { + test('carries passed and failed results forward and runs only what is unfinished, as retries', async () => { + const { consumer, controller, stateDir, options, invocation } = await setUp({ startup: 'fail', persistence: 'hang', uninstall: 'pass' }); + const first = await started(invocation, { + ...options, + onEvent: (event) => { + if (event.scenario === 'persistence' && event.phase === 'steps' && event.status === 'started') controller.abort(); + }, + }); + expect(outcomes(first)).toEqual({ startup: 'failed', persistence: 'cancelled', uninstall: 'not-run' }); + + // The tester fixes whatever made it hang; the scenario now passes. + await rm(consumer.holdPath); + const logBefore = (await calls(consumer)).length; + + const summary = await resumed(first.runId, { ...options, signal: new AbortController().signal }); + + expect(outcomes(summary)).toEqual({ startup: 'failed', persistence: 'passed', uninstall: 'passed' }); + expect(summary.results.find((r) => r.requirement.endsWith('/startup'))?.carried).toBe(true); + expect(summary.exitCode).toBe(1); // the carried failure still counts + expect((await calls(consumer)).slice(logBefore).filter((l) => l.startsWith('steps:')).map((l) => l.split(' ')[0])).toEqual(['steps:persistence', 'steps:uninstall']); + + const state = await readRun(join(stateDir, first.runId)); + const persistenceAttempts = state.attempts.filter((a) => a.requirement.endsWith('/persistence')); + expect(persistenceAttempts.map((a) => a.outcome)).toEqual(['cancelled', 'passed']); + expect(persistenceAttempts[1]?.retryOf).toBe(persistenceAttempts[0]?.id); + expect(state.events.filter((e) => e.type === 'run-started')).toHaveLength(1); + }); + + test('a scenario that was started but never recorded (the process died) becomes an interrupted attempt before it is rerun', async () => { + const { stateDir, options, invocation } = await setUp({ persistence: 'pass' }); + const first = await started(invocation, options); + // Simulate a crash in a later session: a started checkpoint with no attempt after it. + const runDir = join(stateDir, first.runId); + const { appendEvent } = await import('../../src/runner/journal.ts'); + const before = await readRun(runDir); + const last = before.events.at(-1)!; + await appendEvent(runDir, { schemaVersion: 1, id: `${first.runId}.crash`, prev: last.id, recordedAt: '2026-09-23T10:00:00Z', type: 'checkpoint', data: { name: 'scenario-started', requirement: before.attempts[0]!.requirement } }); + + const summary = await resumed(first.runId, options); + + const attempts = (await readRun(runDir)).attempts; + expect(attempts.map((a) => a.outcome)).toEqual(['passed', 'interrupted', 'passed']); + expect(attempts[2]?.retryOf).toBe(attempts[1]?.id); + expect(outcomes(summary)).toEqual({ persistence: 'passed' }); + }); + + test('an unknown run id is refused', async () => { + const { options } = await setUp({ persistence: 'pass' }); + const result = await resumeRun('run-does-not-exist', options); + expect(result.ok).toBe(false); + }); + + test('a candidate that changed since the run started is refused: it would be a different candidate', async () => { + const { consumer, options, invocation } = await setUp({ persistence: 'throw' }); + const first = await started(invocation, options); + await writeFile(join(consumer.dir, 'setup.bin'), 'rebuilt'); + const result = await resumeRun(first.runId, options); + expect(result.ok).toBe(false); + }); + + test('a manifest that now names a different candidate is refused', async () => { + const { consumer, options, invocation } = await setUp({ persistence: 'throw' }); + const first = await started(invocation, options); + const manifest = JSON.parse(await readFile(consumer.candidatePath, 'utf8')) as { id: string }; + await writeFile(consumer.candidatePath, JSON.stringify({ ...manifest, id: 'local-2' })); + const result = await resumeRun(first.runId, options); + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toMatch(/local-1.*local-2|local-2.*local-1/); + }); +}); + +describe('the run\'s exit code', () => { + const r = (...outcomes: RequirementResult['outcome'][]): RequirementResult[] => outcomes.map((outcome, i) => ({ requirement: `windows/s${i}`, outcome })); + + test.each([ + [[], 0], + [['passed', 'passed'], 0], + [['passed', 'manual'], 2], + [['blocked', 'passed'], 2], + [['cancelled', 'blocked'], 3], + [['not-run', 'manual'], 3], + [['interrupted', 'passed'], 3], + // A failure is a verdict on the candidate: it decides the code even when something else was left unfinished. + [['interrupted', 'failed'], 1], + [['failed', 'cancelled', 'not-run', 'blocked', 'manual'], 1], + ] as Array<[RequirementResult['outcome'][], number]>)('%j exits %i', (outcomes, code) => { + expect(exitCodeOf(r(...outcomes))).toBe(code); + }); + + test('a cleanup that failed leaves the environment dirty: 3, even when every scenario passed', () => { + const dirty: RequirementResult = { requirement: 'windows/s0', outcome: 'passed', cleanup: { ok: false, failures: ['app: still-running'] } }; + expect(exitCodeOf([dirty])).toBe(3); + expect(exitCodeOf([dirty, { requirement: 'windows/s1', outcome: 'blocked' }])).toBe(3); + // ...but a failure still decides it. + expect(exitCodeOf([dirty, { requirement: 'windows/s1', outcome: 'failed' }])).toBe(1); + }); +}); + +describe('resuming a run whose journal cannot be trusted', () => { + test('a journal with two different events under one id is refused, and nothing runs', async () => { + const { consumer, stateDir, options, invocation } = await setUp({ persistence: 'throw' }); + const first = await started(invocation, options); + const log = join(stateDir, first.runId, 'events.jsonl'); + const [firstLine] = (await readFile(log, 'utf8')).split('\n'); + const tampered = JSON.parse(firstLine as string) as { data: { machineId: string } }; + tampered.data.machineId = 'machine-someone-else'; + await writeFile(log, `${await readFile(log, 'utf8')}${JSON.stringify(tampered)}\n`); + const before = (await calls(consumer)).length; + + const result = await resumeRun(first.runId, options); + + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toMatch(/inconsistent/); + expect((await calls(consumer)).length).toBe(before); + }); +}); + +describe('resuming checks the candidate is the one the run tested', () => { + test('resume refuses a manifest that keeps the id but now names different bytes: that is a different build', async () => { + const { consumer, options, invocation } = await setUp({ persistence: 'throw' }); + const first = await started(invocation, options); + const { createHash } = await import('node:crypto'); + await writeFile(join(consumer.dir, 'setup.bin'), 'rebuilt'); + const manifest = JSON.parse(await readFile(consumer.candidatePath, 'utf8')) as { artifacts: Array<{ sha256: string }> }; + manifest.artifacts[0]!.sha256 = createHash('sha256').update('rebuilt').digest('hex'); + await writeFile(consumer.candidatePath, JSON.stringify(manifest)); + const before = (await calls(consumer)).length; + + const result = await resumeRun(first.runId, options); + + expect(result.ok).toBe(false); + expect(!result.ok && result.error).toMatch(/different build|bytes/); + expect((await calls(consumer)).length).toBe(before); + }); + + test('a journal that cannot be read at all is a refusal, not a thrown error', async () => { + const { stateDir, options, invocation } = await setUp({ persistence: 'throw' }); + const first = await started(invocation, options); + const log = join(stateDir, first.runId, 'events.jsonl'); + await rm(log); + await (await import('node:fs/promises')).mkdir(log); + const result = await resumeRun(first.runId, options); + expect(result.ok).toBe(false); + }); +}); + +describe('a failed cleanup survives a resume', () => { + test('a passed scenario whose cleanup failed is carried forward with that failure, so the run still exits 3', async () => { + const { consumer, root, options, invocation } = await setUp({ persistence: 'pass', uninstall: 'pass' }); + await writeFile(consumer.failCleanupPath, 'x'); + const first = await started(invocation, options); + // The failed cleanup leaves the root dirty, so the next scenario is blocked. + expect(outcomes(first)).toEqual({ persistence: 'passed', uninstall: 'blocked' }); + expect(first.exitCode).toBe(3); + + // The tester fixes the uninstaller and resets the environment, then resumes. + await rm(consumer.failCleanupPath); + const { runReset } = await import('../../src/cli/environment-commands.ts'); + expect((await runReset(root)).ok).toBe(true); + const summary = await resumed(first.runId, options); + + expect(outcomes(summary)).toEqual({ persistence: 'passed', uninstall: 'passed' }); + const carried = summary.results.find((r) => r.requirement.endsWith('/persistence')); + expect(carried).toMatchObject({ carried: true, cleanup: { ok: false } }); + expect(summary.exitCode).toBe(3); + }); +}); diff --git a/packages/qa/test/fixtures/consumer.ts b/packages/qa/test/fixtures/consumer.ts new file mode 100644 index 0000000..7355990 --- /dev/null +++ b/packages/qa/test/fixtures/consumer.ts @@ -0,0 +1,128 @@ +// Writes a real consumer project to a scratch directory: qa/project.json, a lifecycle module and a scenario file, +// all loadable both by the test worker and by plain `node` (so the CLI executable can run them too). Scenario and +// hook bodies are plain JavaScript in .ts files, which is valid TypeScript and needs no type stripping to speak of. +import { createHash } from 'node:crypto'; +import { mkdir, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { hostProfile, makeTempDir } from './processes.ts'; + +/** + * What a scenario's steps do: pass, fail an assertion, throw, kill the whole process, or `hang`: wait for cancellation + * for as long as the consumer's `holdPath` file exists, and pass once it has been removed. + */ +export type Behaviour = 'pass' | 'fail' | 'throw' | 'hang' | 'crash'; + +const STEPS: Record = { + pass: 'async () => {}', + fail: "async () => { assert.fail('the saved value was not shown after restart'); }", + throw: "async () => { throw new Error('the driver went away'); }", + // An already-aborted signal never fires 'abort' again, so that case must reject straight away. + hang: "(ctx) => !existsSync(HOLD) ? Promise.resolve() : ctx.signal.aborted ? Promise.reject(ctx.signal.reason) : new Promise((_, reject) => ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true }))", + crash: "async () => { process.exit(70); }", +}; + +export interface ConsumerOptions { + /** Scenario id to behaviour, for the host profile. Each becomes an automated requirement in suite "release". */ + scenarios: Record; + /** Manual requirements for the host profile, also in suite "release". */ + manual?: string[]; + /** A scenario id (also listed in `scenarios`) that kills the process the first time it runs and passes afterwards. */ + crashOnce?: string; +} + +export interface Consumer { + dir: string; + projectPath: string; + candidatePath: string; + /** Every hook and scenario call appends ` ` here. */ + logPath: string; + /** While this file exists, `hang` scenarios wait for cancellation. Remove it to let them pass. */ + holdPath: string; + /** While this file exists (it does not by default), the lifecycle's cleanup hook throws. */ + failCleanupPath: string; + profile: string; + artifactBytes: string; +} + +export async function writeConsumer(options: ConsumerOptions): Promise { + // A typo here would otherwise surface as an obscure error deep in generation, or silently test something else. + for (const [id, behaviour] of Object.entries(options.scenarios)) { + if (!Object.hasOwn(STEPS, behaviour)) throw new Error(`writeConsumer: scenario "${id}" has unknown behaviour "${behaviour}"`); + } + if (options.crashOnce !== undefined && !Object.hasOwn(options.scenarios, options.crashOnce)) { + throw new Error(`writeConsumer: crashOnce "${options.crashOnce}" is not one of the scenarios`); + } + const dir = await makeTempDir('qa-consumer-'); + const qa = join(dir, 'qa'); + await mkdir(qa, { recursive: true }); + const profile = hostProfile(); + const logPath = join(dir, 'calls.log'); + const ids = Object.keys(options.scenarios); + const manual = options.manual ?? []; + + await writeFile(join(dir, 'package.json'), JSON.stringify({ name: 'consumer', private: true, type: 'module' })); + await writeFile( + join(qa, 'project.json'), + JSON.stringify({ + schemaVersion: 1, + projectId: 'consumer', + releaseBranch: 'main', + profiles: [profile], + requirements: [ + ...ids.map((id) => ({ key: `${profile.id}/${id}`, mode: 'automated', title: id, capabilities: [] })), + ...manual.map((id) => ({ key: `${profile.id}/${id}`, mode: 'manual', title: id, capabilities: [] })), + ], + suites: [{ id: 'release', requirements: [...ids, ...manual].map((id) => `${profile.id}/${id}`) }], + scenarioFiles: ['scenarios.ts'], + lifecycleModule: 'lifecycle.ts', + workflows: { prepare: 'qa-prepare.yml', gate: 'qa-gate.yml', publish: 'qa-publish.yml' }, + markers: { releaseNotes: 'release-notes', qa: 'qa' }, + }), + ); + + const log = (phase: string) => `appendFileSync(${JSON.stringify(logPath)}, \`${phase} \${ctx.artifact ? ctx.artifact.path : '-'}\\n\`)`; + const failCleanupPath = join(dir, 'fail-cleanup'); + await writeFile( + join(qa, 'lifecycle.ts'), + [ + "import { appendFileSync, existsSync } from 'node:fs';", + 'export const lifecycle = {', + ...['install', 'reset', 'launch'].map((phase) => ` ${phase}: async (ctx) => { ${log(phase)}; },`), + ` cleanup: async (ctx) => { ${log('cleanup')}; if (existsSync(${JSON.stringify(failCleanupPath)})) throw new Error('the uninstaller crashed'); },`, + '};', + ].join('\n'), + ); + + const crashMarker = join(dir, 'crashed-once'); + const holdPath = join(dir, 'hold'); + await writeFile(holdPath, 'x'); + await writeFile( + join(qa, 'scenarios.ts'), + [ + "import assert from 'node:assert';", + "import { appendFileSync, existsSync, writeFileSync } from 'node:fs';", + 'export const scenarios = [', + ...ids.map((id) => { + const behaviour = options.crashOnce === id + ? `async () => { if (!existsSync(${JSON.stringify(crashMarker)})) { writeFileSync(${JSON.stringify(crashMarker)}, 'x'); process.exit(70); } }` + : STEPS[options.scenarios[id] as Behaviour].replace('HOLD', JSON.stringify(holdPath)); + return ` { id: ${JSON.stringify(id)}, steps: async (ctx) => { ${log(`steps:${id}`)}; return (${behaviour})(ctx); } },`; + }), + '];', + ].join('\n'), + ); + + const artifactBytes = 'installer bytes'; + await writeFile(join(dir, 'setup.bin'), artifactBytes); + const candidatePath = join(dir, 'candidate.json'); + await writeFile( + candidatePath, + JSON.stringify({ + schemaVersion: 1, + id: 'local-1', + artifacts: [{ profile: profile.id, name: 'setup.bin', path: 'setup.bin', sha256: createHash('sha256').update(artifactBytes).digest('hex') }], + }), + ); + + return { dir, projectPath: join(qa, 'project.json'), candidatePath, logPath, holdPath, failCleanupPath, profile: profile.id, artifactBytes }; +} diff --git a/packages/qa/test/fixtures/processes.ts b/packages/qa/test/fixtures/processes.ts index ee09e0e..602ba97 100644 --- a/packages/qa/test/fixtures/processes.ts +++ b/packages/qa/test/fixtures/processes.ts @@ -20,7 +20,7 @@ export function startUnrelatedProcess(script = 'setInterval(() => {}, 1000)'): C } /** Registers a process started some other way (for example by spawnOwned) so it is stopped after the test. */ -export function trackProcess(child: ChildProcess): ChildProcess { +export function trackProcess(child: T): T { child.on('error', () => undefined); started.push(child); return child; diff --git a/packages/qa/test/model/local-candidate.test.ts b/packages/qa/test/model/local-candidate.test.ts new file mode 100644 index 0000000..da200b3 --- /dev/null +++ b/packages/qa/test/model/local-candidate.test.ts @@ -0,0 +1,85 @@ +import { describe, expect, test } from 'vitest'; +import { parseLocalCandidate } from '../../src/model/local-candidate.ts'; + +const valid = () => ({ + schemaVersion: 1, + id: 'local-2026-09-23.1', + artifacts: [ + { profile: 'windows', name: 'setup.exe', path: 'dist/setup.exe', sha256: 'b'.repeat(64) }, + { profile: 'linux', name: 'app.deb', path: 'dist/app.deb', sha256: 'c'.repeat(64) }, + ], +}); + +const issues = (input: unknown): string[] => { + const result = parseLocalCandidate(input); + if (result.ok) throw new Error('expected a validation failure'); + return result.error.issues.map((i) => `${i.path}:${i.code}`); +}; + +describe('a local candidate manifest', () => { + test('a valid manifest is accepted as written', () => { + expect(parseLocalCandidate(valid())).toEqual({ ok: true, value: valid() }); + }); + + test('another schema version is refused without looking further', () => { + expect(issues({ ...valid(), schemaVersion: 2 })).toEqual(['schemaVersion:unknown-schema-version']); + }); + + test('at least one artifact is required', () => { + expect(issues({ ...valid(), artifacts: [] })).toContain('artifacts:empty'); + }); + + test('two artifacts for the same profile are refused: which one to test would be ambiguous', () => { + const [first] = valid().artifacts; + expect(issues({ ...valid(), artifacts: [first, { ...first, name: 'other.exe' }] })).toContain('artifacts[1].profile:duplicate'); + }); + + test.each([ + ['an absolute path', '/etc/passwd'], + ['a path that climbs out of the manifest directory', '../outside/setup.exe'], + ['a Windows drive path', 'C:/setup.exe'], + ])('%s is refused', (_label, path) => { + const [first] = valid().artifacts; + expect(issues({ ...valid(), artifacts: [{ ...first, path }] })).toContain('artifacts[0].path:unsafe-path'); + }); + + test('a malformed hash is refused', () => { + const [first] = valid().artifacts; + expect(issues({ ...valid(), artifacts: [{ ...first, sha256: 'B'.repeat(64) }] })).toContain('artifacts[0].sha256:malformed-hash'); + }); + + test('an unknown field is refused rather than ignored', () => { + expect(issues({ ...valid(), note: 'x' })).toContain('note:unknown-field'); + }); +}); + +describe('what an artifact entry must be', () => { + const entry = (overrides: Record) => ({ ...valid(), artifacts: [{ ...valid().artifacts[0], ...overrides }] }); + + test.each([ + ['a non-object entry', { ...valid(), artifacts: ['setup.exe'] }, 'artifacts[0]:invalid-type'], + ['a name that is a path', entry({ name: 'dist/setup.exe' }), 'artifacts[0].name:unsafe-path'], + ['a Windows device name', entry({ name: 'CON' }), 'artifacts[0].name:unsafe-path'], + ['an empty name', entry({ name: '' }), 'artifacts[0].name:empty'], + ['an empty path', entry({ path: '' }), 'artifacts[0].path:empty'], + ['an upper-case profile', entry({ profile: 'Windows' }), 'artifacts[0].profile:malformed-id'], + ])('%s is refused', (_label, input, expected) => { + expect(issues(input)).toContain(expected); + }); + + test.each([['id'], ['artifacts']])('a missing %s is refused', (field) => { + const input: Record = valid(); + delete input[field]; + expect(issues(input)).toContain(`${field}:missing-field`); + }); + + test.each([['profile'], ['name'], ['path'], ['sha256']])('an artifact without %s is refused', (field) => { + const artifact: Record = { ...valid().artifacts[0] }; + delete artifact[field]; + expect(issues({ ...valid(), artifacts: [artifact] })).toContain(`artifacts[0].${field}:missing-field`); + }); + + test.each([[null], ['a string'], [[]]])('%j is not a manifest at all', (input) => { + expect(parseLocalCandidate(input).ok).toBe(false); + }); +}); diff --git a/packages/qa/test/runner/execute-hardening.test.ts b/packages/qa/test/runner/execute-hardening.test.ts index c0d134a..622eaee 100644 --- a/packages/qa/test/runner/execute-hardening.test.ts +++ b/packages/qa/test/runner/execute-hardening.test.ts @@ -112,7 +112,8 @@ describe('every wait is bounded, including prerequisites and the event sink', () const { context } = await arrange({ lifecycle: lifecycleOf(calls), emit: (event) => (event.phase === 'install' && event.status === 'started' ? never() : undefined), - timeouts: { phaseMs: 80, stepsMs: 2000, cleanupMs: 2000 }, + // Shared with prerequisites: enough room to reach install on a loaded machine, where the sink then hangs. + timeouts: { phaseMs: 300, stepsMs: 2000, cleanupMs: 2000 }, }); const result = await executeScenario(context, scenarioOf()); expect(result).toMatchObject({ outcome: 'interrupted', reason: 'infrastructure-error' }); @@ -165,13 +166,16 @@ describe('hooks that outlive their phase', () => { expect((await executeScenario({ ...context, signal: new AbortController().signal }, scenarioOf())).reason).toBe('dirty-environment'); }); + // The phase budget is shared with prerequisites, so it must leave room to reach install on a loaded machine; + // otherwise prerequisites time out instead and this passes without testing an install being cut off at all. test('a hook that stops soon after being cut off does not dirty the environment', async () => { const { context } = await arrange({ - lifecycle: lifecycleOf([], { install: () => sleep(80) }), - timeouts: { phaseMs: 20, stepsMs: 2000, cleanupMs: 2000, abandonedGraceMs: 1000 }, + lifecycle: lifecycleOf([], { install: () => sleep(900) }), + timeouts: { phaseMs: 300, stepsMs: 2000, cleanupMs: 2000, abandonedGraceMs: 3000 }, }); const result = await executeScenario(context, scenarioOf()); expect(result).toMatchObject({ outcome: 'interrupted', reason: 'timeout' }); + expect(result.detail).toContain('install'); expect(result.cleanup.ok).toBe(true); }); diff --git a/packages/qa/test/runner/execute.test.ts b/packages/qa/test/runner/execute.test.ts index 23492d3..0634a0c 100644 --- a/packages/qa/test/runner/execute.test.ts +++ b/packages/qa/test/runner/execute.test.ts @@ -15,6 +15,41 @@ vi.setConfig({ testTimeout: 30_000 }); afterEach(cleanUpProcessesAndRoots); const sleeper = ['-e', 'setInterval(() => {}, 1000)']; +// Spawning and reaping a real process each read its identity, which on Windows starts PowerShell and can take seconds +// under load, so the steps (spawn) and cleanup (reap) budgets both allow for it. A cleanup budget too tight for that +// cuts reaping off; it then finishes in the background, after the assertions below have looked. +const REAPING_MS = 20_000; + +describe('what the hooks are told about the candidate', () => { + test('every hook and the steps receive the same verified artifact and the candidate identity', async () => { + const seen: string[] = []; + const artifact = { name: 'setup.exe', path: '/tmp/setup.exe', sha256: 'a'.repeat(64) }; + const note = (phase: string) => async (ctx: { artifact?: typeof artifact; candidate: { id: string } }) => { + seen.push(`${phase}:${ctx.candidate.id}:${ctx.artifact?.path}`); + }; + const { context } = await arrange({ + candidate: { id: 'local-7' }, + artifact, + lifecycle: { install: note('install'), reset: note('reset'), launch: note('launch'), cleanup: note('cleanup') }, + }); + await executeScenario(context, scenarioOf({ steps: note('steps') })); + expect(seen).toEqual(['install', 'reset', 'launch', 'steps', 'cleanup'].map((phase) => `${phase}:local-7:/tmp/setup.exe`)); + }); + + test('the artifact is simply absent when the run has none', async () => { + let seen: unknown = 'unset'; + const { context } = await arrange(); + await executeScenario(context, scenarioOf({ steps: async (ctx) => { seen = ctx.artifact; } })); + expect(seen).toBeUndefined(); + }); + + test('a full GitHub candidate record is still accepted as the candidate', async () => { + let id = ''; + const { context } = await arrange({ candidate: candidate() }); + await executeScenario(context, scenarioOf({ steps: async (ctx) => { id = ctx.candidate.id; } })); + expect(id).toBe(candidate().id); + }); +}); describe('a passing run', () => { test('runs the phases in order, emits an event for each, and cleans up', async () => { @@ -292,7 +327,7 @@ describe('cancelled', () => { describe('owning only what the run created', () => { test('stops a process the scenario spawned even if the cleanup hook forgot it, and leaves an unrelated process alone', async () => { - const { context, testRoot } = await arrange(); + const { context, testRoot } = await arrange({ timeouts: { phaseMs: 2000, stepsMs: REAPING_MS, cleanupMs: REAPING_MS } }); const unrelated = startUnrelatedProcess(); let spawnedPid = 0; @@ -304,13 +339,14 @@ describe('owning only what the run created', () => { })); expect(result.outcome).toBe('passed'); + expect(result.cleanup).toEqual({ ok: true, failures: [] }); await eventually(() => !isAlive(spawnedPid)); expect(isAlive(unrelated.pid as number)).toBe(true); expect(await readLedger(testRoot)).toEqual([]); }); test('reaps what the run owned even when the steps failed', async () => { - const { context } = await arrange(); + const { context } = await arrange({ timeouts: { phaseMs: 2000, stepsMs: REAPING_MS, cleanupMs: REAPING_MS } }); let spawnedPid = 0; const result = await executeScenario(context, scenarioOf({ steps: async (ctx) => { @@ -319,6 +355,7 @@ describe('owning only what the run created', () => { }, })); expect(result.outcome).toBe('failed'); + expect(result.cleanup).toEqual({ ok: true, failures: [] }); await eventually(() => !isAlive(spawnedPid)); });