feat(cli): run and resume a suite from a checkout (Task 2.2, part 3a) #10
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Decision: running a suite from a checkout (Task 2.2, part 3a) | ||
|
|
||
| Status: **implemented** for the CLI's `run`, `resume` and `reset`, against fixture consumer projects. Running the real sample (`examples/tauri-smoke`) through a Tauri driver adapter is part 3b. | ||
|
|
||
| ## The local candidate manifest | ||
|
|
||
| `run --candidate <file>` takes a local manifest, not a GitHub candidate record: | ||
|
|
||
| ```json | ||
| { | ||
| "schemaVersion": 1, | ||
| "id": "local-2026-09-23.1", | ||
| "artifacts": [ | ||
| { "profile": "windows", "name": "setup.exe", "path": "dist/setup.exe", "sha256": "<64 lower-case hex>" } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| - One artifact per profile; `path` is relative to the manifest's own directory and cannot leave it, lexically or through a link: the real location must be inside the manifest's real directory. | ||
| - Before anything is installed, the chosen profile's file is hashed at its real location and must match `sha256`; the location is re-checked after hashing. Other profiles' files are never read. | ||
| - It has no source or build provenance, so it can never stand in for a GitHub candidate at the merge gate (Stage 3). Hooks receive only `candidate.id` and the verified `artifact` (`name`, its real absolute `path` with no link left in it, `sha256`); a full GitHub candidate record also fits `candidate`. | ||
|
|
||
| ## Consumer code | ||
|
|
||
| `qa/project.json` names a lifecycle module exporting `lifecycle` (`install`, `reset`, `launch`, `cleanup`) and scenario files exporting `scenarios` (`{ id, setup?, steps }[]`). A requirement key `<profile>/<id>` runs the scenario with that id. Running executes the project's code; the CLI does it only for a project the user points it at, and every problem (missing hook, undefined scenario, duplicate id, a module that throws while loading) is reported before anything is installed. | ||
|
|
||
| ## State | ||
|
|
||
| - Default test root: `.release-qa` under the current directory; default state directory: `.release-qa/runs` (both gitignored). Each run is `runs/<run id>/` with `invocation.json` (what was asked, absolute paths, and the tested artifact's SHA-256), `events.jsonl` (the Task 1.3 journal) and `summary.json`. | ||
| - The machine is identified in run records by a random token kept in the state directory, never the host name. | ||
| - `run` announces `run <id> started` on stderr before anything runs, in every output mode, with the `resume` command to use (including a custom `--state`, quoted so it pastes safely in bash and PowerShell), so a run can be resumed even if the process dies. | ||
| - A run id is a run id, never a path: `resume --run` accepts only the journal's id grammar, which has no path separators (`/`, `\`, `:`). | ||
|
|
||
| ## Journal and resume | ||
|
|
||
| Each scenario records a `scenario-started` checkpoint, then an attempt, then a `cleanup-failed` checkpoint if its cleanup failed. `resume --run <id>`: | ||
|
|
||
| - re-verifies the candidate: the same manifest id, and bytes whose SHA-256 is the one recorded when the run started (a manifest edited to name new bytes under the same id is refused: that is a different build); | ||
| - carries **passed** and **failed** forward (a failure cannot disappear by being run again), with any recorded cleanup failure; | ||
| - reruns anything else (blocked, cancelled, interrupted, never reached) as a retry (`retryOf`) of its latest attempt; | ||
| - turns a `scenario-started` with no attempt after it (the process died) into an explicit `interrupted` attempt first, so the crash stays in the run's history. | ||
|
|
||
| A run whose journal has conflicting or cyclic events, or cannot be read, is not continued. | ||
|
|
||
| ## Outcomes and exit codes | ||
|
|
||
| One scenario failing does not stop the others; cancellation stops the running scenario (its cleanup still runs) and starts nothing further (`not-run`). Exit codes, highest rule first: any **failed** → `1`; any interrupted, cancelled or not-run, or any cleanup that failed (the environment was left dirty, however the scenario went) → `3`; any blocked or manual → `2`; otherwise `0`. Configuration and verification problems are `3` before anything runs. | ||
|
|
||
| ## Reset | ||
|
|
||
| A crash can leave owned resources and a dirty marker in the test root, which blocks later runs. `reset` reaps the ledger and clears the marker only when that fully succeeds; it refuses an undesignated root and a root a run currently holds. | ||
|
|
||
| ## Not verified | ||
|
|
||
| - Cancellation by a real signal is tested on Linux only (CI). On Windows a console Ctrl+C reaches the same handler, but a test cannot send one to another process. | ||
| - Evidence files (screenshots, logs) are not collected yet; attempts record `evidence: []`. | ||
| - The artifact's contents could still change after verification; only copying it into the run's own storage would close that. Links cannot redirect it, and the install hook consumes it at once. | ||
| - Consumer code runs in the CLI's own process: a scenario that calls `process.exit` takes the CLI with it. That is the crash `resume` recovers from, not something prevented. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import { createHash } from 'node:crypto'; | ||
| import { createReadStream } from 'node:fs'; | ||
| import { readFile, realpath, stat } from 'node:fs/promises'; | ||
| import { dirname, resolve, sep } from 'node:path'; | ||
| import { parseLocalCandidate } from '../model/local-candidate.ts'; | ||
| import type { ArtifactRef, CandidateRef } from '../runner/execute.ts'; | ||
|
|
||
| export type LoadedCandidate = { ok: true; candidate: CandidateRef; artifact: ArtifactRef } | { ok: false; error: string }; | ||
|
|
||
| const message = (error: unknown): string => (error instanceof Error ? error.message : String(error)); | ||
| const fold = (path: string): string => (process.platform === 'win32' ? path.toLowerCase() : path); | ||
| const isInside = (directory: string, path: string): boolean => fold(path).startsWith(fold(directory.endsWith(sep) ? directory : directory + sep)); | ||
|
|
||
| /** | ||
| * Reads a local candidate manifest, picks the artifact for `profile`, and checks the file's bytes against the | ||
| * manifest's SHA-256 before anything is installed. Never throws. Only the chosen profile's file is read. | ||
| */ | ||
| export async function loadCandidate(manifestPath: string, profile: string): Promise<LoadedCandidate> { | ||
| let text: string; | ||
| try { | ||
| text = await readFile(manifestPath, 'utf8'); | ||
| } catch (error) { | ||
| return { ok: false, error: `could not read the candidate manifest ${manifestPath}: ${message(error)}` }; | ||
| } | ||
| let parsed: unknown; | ||
| try { | ||
| parsed = JSON.parse(text); | ||
| } catch (error) { | ||
| return { ok: false, error: `${manifestPath} is not valid JSON: ${message(error)}` }; | ||
| } | ||
| const result = parseLocalCandidate(parsed); | ||
| if (!result.ok) return { ok: false, error: `${manifestPath}: ${result.error.message}` }; | ||
| const manifest = result.value; | ||
|
|
||
| const chosen = manifest.artifacts.find((a) => a.profile === profile); | ||
| if (chosen === undefined) { | ||
| return { ok: false, error: `candidate "${manifest.id}" has no artifact for profile "${profile}"; it has: ${manifest.artifacts.map((a) => a.profile).join(', ')}` }; | ||
| } | ||
|
|
||
| // Relative to the manifest, not to wherever the command happens to be run from. | ||
| const path = resolve(dirname(manifestPath), ...chosen.path.split('/')); | ||
| const info = await stat(path).catch(() => undefined); | ||
| if (info === undefined) return { ok: false, error: `the artifact ${path} for profile "${profile}" does not exist` }; | ||
| if (!info.isFile()) return { ok: false, error: `the artifact ${path} for profile "${profile}" is not a file` }; | ||
| // The manifest's path check is lexical; a link on the way could still lead elsewhere. Judge the real locations, then | ||
| // hash the real file and give hooks that path, so no link is left in it that could be swapped to point elsewhere. | ||
| let realDirectory: string; | ||
| let realFile: string; | ||
| try { | ||
| [realDirectory, realFile] = await Promise.all([realpath(dirname(manifestPath)), realpath(path)]); | ||
| } catch (error) { | ||
| return { ok: false, error: `could not resolve the artifact ${path}: ${message(error)}` }; | ||
| } | ||
| if (!isInside(realDirectory, realFile)) { | ||
| return { ok: false, error: `the artifact ${path} for profile "${profile}" leads outside the manifest's directory, to ${realFile}` }; | ||
| } | ||
|
|
||
| let actual: string; | ||
| try { | ||
| actual = await sha256Of(realFile); | ||
| // A link swapped while the file was being hashed would make the check above describe some other file. | ||
| if ((await realpath(path)) !== realFile) return { ok: false, error: `the artifact ${path} changed location while it was being verified` }; | ||
|
There was a problem hiding this comment. P3: The new link-swap refusal ("changed location while it was being verified") has no test. The added tests only cover a Prompt for AI agents
Member
Author
There was a problem hiding this comment. Fixed in b7143e3. The new test mocks realpath so the artifact resolves to its real location before hashing and somewhere else afterwards, and asserts the 'changed location' refusal and that the re-check ran. I confirmed it fails when the re-check is removed. |
||
| } catch (error) { | ||
| return { ok: false, error: `could not read the artifact ${path}: ${message(error)}` }; | ||
| } | ||
| if (actual !== chosen.sha256) { | ||
| return { ok: false, error: `the artifact ${path} does not match the candidate: expected SHA-256 ${chosen.sha256}, found ${actual}` }; | ||
| } | ||
| return { ok: true, candidate: { id: manifest.id }, artifact: { name: chosen.name, path: realFile, sha256: actual } }; | ||
| } | ||
|
|
||
| /** Streams the file, so an installer of any size is hashed without being held in memory. */ | ||
| function sha256Of(path: string): Promise<string> { | ||
| return new Promise((resolveHash, rejectHash) => { | ||
| const hash = createHash('sha256'); | ||
| createReadStream(path) | ||
| .on('data', (chunk) => hash.update(chunk)) | ||
| .on('error', rejectHash) | ||
| .on('end', () => resolveHash(hash.digest('hex'))); | ||
| }); | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.