feat(cli): run and resume a suite from a checkout (Task 2.2, part 3a)

[Andreas-Froyland]

Summary

Task 2.2 of the plan, part 3a: running a suite from the CLI.

node packages/qa/src/cli/main.ts run --project <qa/project.json> --candidate <candidate.json> --profile <id> --suite <id> [--root] [--state] [--json]
node packages/qa/src/cli/main.ts resume --run <run id> [--state] [--json]
node packages/qa/src/cli/main.ts reset [--root] [--json]

Part 3b (next PR) adds the Tauri driver adapter and the examples/tauri-smoke consumer, and runs the real sample on Windows and Ubuntu/Xvfb, which is what meets the plan's exit criterion. This PR is tested end to end against generated fixture consumers, including through the real executable. Full design notes: docs/decisions/local-runs.md.

What run does

1. Checks everything that doesn't touch the machine first: project, plan, candidate bytes, and the consumer's lifecycle and scenario modules. A mistake is reported, with exit 3, before any run state exists or anything is installed.
2. Candidate: reads a local manifest ({ schemaVersion, id, artifacts: [{ profile, name, path, sha256 }] }), picks this profile's artifact (path relative to the manifest), and hashes the file. A mismatch names both hashes. Other profiles' files are never read. The manifest carries no provenance, so it can never stand in for a GitHub candidate at the gate.
3. Hooks now receive candidate as an identity (a full GitHub candidate record still fits) plus the verified artifact (name, absolute path, sha256), as the design spec says they should.
4. Journal (Task 1.3): records run-started, then per scenario a scenario-started checkpoint and an attempt, chained by prev. The machine id is a random token kept in the state directory, never the host name. The run id is announced on stderr before anything runs, so a run can be resumed even if the process dies.
5. Outcomes: one scenario failing doesn't stop the others. Cancellation (SIGINT/SIGTERM) stops the running scenario, still runs its cleanup, and starts nothing further (not-run). A second interrupt exits at once.

resume

• Re-verifies the candidate: same manifest id and same bytes, or it refuses.
• Carries passed and failed forward. A failure can't disappear by being run again.
• Reruns everything else as a retry (retryOf) of its latest attempt.
• A scenario-started with no attempt after it means the process died. That becomes an explicit interrupted attempt first, so the crash stays in history.
• Refuses a journal with conflicting or cyclic events.

reset

A crash can leave owned resources and a dirty marker behind, which blocks later runs. reset reaps the ledger and clears the marker only when that fully succeeds. It refuses an undesignated root, and a root a run currently holds.

Exit codes

The highest rule that applies wins:

Code	Meaning
1	any scenario failed (a verdict on the candidate)
3	otherwise, anything interrupted, cancelled or not-run; also configuration or verification problems
2	otherwise, anything blocked or a manual check left
0	everything passed

Also in this PR: timing-sensitive executor tests

This PR's executable tests spawn extra Node processes, and the added load exposed two weak spots in existing tests. Both are fixed here:

• Reaping a real process. The two tests that do this used the fixture's 2 s cleanup budget. On Windows, reaping each process starts a PowerShell identity lookup, which can take seconds. When the budget runs out, reaping is cut off and finishes in the background, so the tests passed or failed on timing. I confirmed this by forcing a 50 ms budget: result.cleanup reported the cut-off and the test still passed. They now use 20 s and assert result.cleanup succeeded, so a cut-off fails with its own message. My earlier memory note had misdiagnosed this as a ledger write race; that was wrong, because executeScenario awaits the rewrite.
• Tiny shared phase budgets. Two tests shared a 20 ms and an 80 ms phase budget with prerequisites. Under load, prerequisites used up the budget first. The event-sink test then failed, and the hook-cut-off test passed without testing what it names. Both now have 300 ms, and the cut-off test asserts it was install that was cut off.

Testing

Written test-first. Coverage:

• the manifest schema
• candidate loading and hashing
• plan selection
• consumer loading (missing hooks, undefined or duplicate scenarios, modules that throw)
• run and resume, in-process
• main exit codes
• the executable: a missing candidate, a genuine process.exit in the middle of a scenario followed by resume, and SIGINT cancellation

Mutation-checked 22 rules with a throwaway script. Two survived: exit-code precedence when a run has a failure and an interruption together, and resuming an inconsistent journal. Both were closed with new tests and reconfirmed killed. Files were verified restored byte for byte and the script was deleted.

• Clean npm ci, npm run typecheck, npm test: 639 passed, 3 skipped (Windows 11, Node 24.13). No leaked processes, temp directories, home markers or stray .release-qa state.
• CI on ubuntu-24.04 and windows-2025, where the Linux job runs the real SIGINT cancellation and the mid-scenario crash followed by resume through the executable.

Review rounds

cubic raised 23 comments over three rounds. 22 were fixed test-first; one, a claim of parallel execution that nothing makes, was declined with reasons in its thread. The main fixes:

• resume is tied to the tested bytes: the artifact's SHA-256 is kept at start, and a rebuild under the same id is refused.
• A failed cleanup makes the run exit 3, and it survives resume: it is journaled as a cleanup-failed checkpoint and resurfaced on the carried result.
• --run can't be a path.
• Artifact paths can't leave the manifest directory through a link. Hooks get the file's real, link-free path, and a link swapped during hashing is refused.
• Consumer-module getters are inspected inside the loader's error boundary.
• The resume hint pastes safely in bash and PowerShell.

Each round's new rules were mutation-checked: 7 and 6 mutants, all killed, plus the round-3 re-check test verified to fail without its check.

Not verified

• SIGINT cancellation through the executable is tested on Linux only. Node on Windows can't send SIGINT to another process (kill() terminates it outright). A console Ctrl+C reaches the same handler there, but a test can't send one. The in-process cancellation path is tested on both OSes.
• No evidence files yet. Attempts record evidence: []; screenshots and logs come with the driver adapter.
• Consumer code runs in the CLI's own process. A scenario that calls process.exit takes the CLI down with it, which is exactly the crash resume handles. Isolating consumer code in a child process is not done.

🤖 Generated with Claude Code

---

Summary by cubic

Adds run, resume, and reset so a suite can be run and resumed from a checkout against a locally built candidate, verified by SHA-256 before anything is installed. The CLI previously stopped at doctor, designate, and status; it now executes real scenarios and records them in the durable run journal.

• run checks the project, plan, candidate, and consumer modules before touching the machine, and refuses a suite with nothing to do for the profile.
• The local candidate manifest lists one artifact per profile with a relative path that can't leave its directory; the artifact is hashed at its real path, the resolution is re-checked after hashing so a swapped link is refused, and only the chosen profile's file is read.
• Hooks receive the candidate identity and the verified artifact (name, real path, sha256), so the path they get has no link left in it to swap.
• The journal records each scenario start and its attempt, chained by prev; the machine id is a random token, never the hostname, and the run id is announced on stderr before anything runs.
• One failing scenario doesn't stop the others. SIGINT/SIGTERM cancels the running scenario, still runs its cleanup, and a second interrupt exits at once.
• resume re-verifies the manifest id and the artifact's SHA-256 — both kept with the invocation — carries passed and failed forward (a failure can't be erased by rerunning), reruns everything else as retries, turns a started-but-unrecorded scenario into an explicit interrupted attempt, and refuses journals with conflicting or cyclic events. A recorded cleanup failure is journaled and resurfaced on the carried result so the run keeps exit 3. The run id must match the journal's id grammar so it can't leave the state directory, and both consumer loading and journal reads happen inside error boundaries so a throwing getter or unexpected rejection is a refusal, not a crash.
• reset clears the dirty marker only when reaping fully succeeds, and refuses an undesignated root or one a run currently holds.
• Exit codes: 1 any scenario failed; otherwise 3 for anything interrupted, cancelled, not-run, a failed cleanup, or a configuration/verification problem; 2 for blocked items or manual checks left; 0 all passed.
• Consumer code runs in the CLI's own process, so a scenario calling process.exit takes the CLI down too — the crash resume handles. Attempts record no evidence files yet; those come with the driver adapter.

Bug Fixes

• Fixed two timing-sensitive executor tests that flaked under the new process-spawning tests' load: process reaping now has a 20 s cleanup budget and asserts it succeeded, and shared phase budgets were raised so prerequisites can't exhaust them.

^{Written for commit b7143e3. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 25 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/qa/src/cli/run.ts">

<violation number="1" location="packages/qa/src/cli/run.ts:147">
P2: This loop awaits every scenario before starting the next, so multi-scenario suites are always serialized and cannot provide the advertised parallel execution. Implement the intended concurrent scheduling with safe per-scenario resource/journal handling, or remove that feature claim.</violation>
</file>

<file name="packages/qa/src/cli/main.ts">

<violation number="1" location="packages/qa/src/cli/main.ts:122">
P1: A scenario can pass its steps while cleanup fails, leaving the test root dirty; this returns exit code 0 because `summary.exitCode` ignores cleanup failures. Treat any cleanup failure as `EXIT.infrastructure` while preserving scenario-failure precedence.</violation>
</file>

<file name="packages/qa/test/cli/main.test.ts">

<violation number="1" location="packages/qa/test/cli/main.test.ts:216">
P3: If `await eventually(...)` throws, `controller.abort()` never runs and `running` is never awaited, so the hang scenario's pending promise keeps the run alive and the test fails only via the 30s test timeout instead of the assertion that was violated. Wrap the poll/abort/await sequence in try/finally: abort and await `running` in the finally so a pre-condition failure still cancels the run and resolves the promise, then rethrow.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: A scenario can pass its steps while cleanup fails, leaving the test root dirty; this returns exit code 0 because summary.exitCode ignores cleanup failures. Treat any cleanup failure as EXIT.infrastructure while preserving scenario-failure precedence.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/src/cli/main.ts, line 122:

<comment>A scenario can pass its steps while cleanup fails, leaving the test root dirty; this returns exit code 0 because `summary.exitCode` ignores cleanup failures. Treat any cleanup failure as `EXIT.infrastructure` while preserving scenario-failure precedence.</comment>

<file context>
@@ -67,11 +78,68 @@ export async function main(argv: readonly string[], io: Io = defaultIo, cwd: ()
+        return EXIT.infrastructure;
+      }
+      printRun(io, command.json, result.summary);
+      return result.summary.exitCode;
+    }
   }
</file context>

Suggested change

	return result.summary.exitCode;
	return result.summary.results.some((r) => r.cleanup !== undefined && !r.cleanup.ok) ? (result.summary.exitCode === EXIT.scenarioFailure ? EXIT.scenarioFailure : EXIT.infrastructure) : result.summary.exitCode;

[Andreas-Froyland]

Fixed in 488e47d. A failed cleanup now makes the run exit 3, even if every scenario passed; a failure still decides 1. Tests in the exit-code table.

[cubic-dev-ai]

P2: This loop awaits every scenario before starting the next, so multi-scenario suites are always serialized and cannot provide the advertised parallel execution. Implement the intended concurrent scheduling with safe per-scenario resource/journal handling, or remove that feature claim.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/src/cli/run.ts, line 147:

<comment>This loop awaits every scenario before starting the next, so multi-scenario suites are always serialized and cannot provide the advertised parallel execution. Implement the intended concurrent scheduling with safe per-scenario resource/journal handling, or remove that feature claim.</comment>

<file context>
@@ -0,0 +1,300 @@
+  let attemptCount = state.attempts.length;
+  const newAttemptId = (): string => `${runId}.a${++attemptCount}`;
+
+  for (const scenario of prepared.consumer.scenarios) {
+    const key = scenario.requirement.key;
+    const history = historyOf(state, key);
</file context>

[Andreas-Froyland]

Not changing this: nothing claims parallel execution. I checked the code, README, docs/decisions/local-runs.md, the plan and the design spec, and none of them mention it. Scenarios in one run are serial on purpose: they share one test root, which only one run may hold at a time (the Task 2.1 lock), and each scenario's reset/install/cleanup would interfere with another's running in the same root.

[cubic-dev-ai]

P3: If await eventually(...) throws, controller.abort() never runs and running is never awaited, so the hang scenario's pending promise keeps the run alive and the test fails only via the 30s test timeout instead of the assertion that was violated. Wrap the poll/abort/await sequence in try/finally: abort and await running in the finally so a pre-condition failure still cancels the run and resolves the promise, then rethrow.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/test/cli/main.test.ts, line 216:

<comment>If `await eventually(...)` throws, `controller.abort()` never runs and `running` is never awaited, so the hang scenario's pending promise keeps the run alive and the test fails only via the 30s test timeout instead of the assertion that was violated. Wrap the poll/abort/await sequence in try/finally: abort and await `running` in the finally so a pre-condition failure still cancels the run and resolves the promise, then rethrow.</comment>

<file context>
@@ -149,3 +154,87 @@ describe('designate and status', () => {
+    await designated(consumer.dir);
+    const controller = new AbortController();
+    const out = io();
+    const running = main([...runArgs(consumer), '--json'], out.sink, () => consumer.dir, controller.signal);
+    await eventually(async () => (await readFile(consumer.logPath, 'utf8').catch(() => '')).includes('steps:persistence'));
+    controller.abort();
</file context>

[Andreas-Froyland]

Fixed in 488e47d. The abort is now in a finally, so a failed wait still cancels and settles the run.

[cubic-dev-ai]

All reported issues were addressed across 16 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 10 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/qa/src/cli/candidate.ts">

<violation number="1" location="packages/qa/src/cli/candidate.ts:62">
P3: The new link-swap refusal ("changed location while it was being verified") has no test. The added tests only cover a `realpath` that throws, a link leading outside the manifest directory, and a link inside it being resolved — none trigger this branch, which is the main new defense against a link swapped mid-verification. Add a test that swaps the directory link between two targets (or mocks `realpath` to return a different path on the re-check call) and asserts the `changed location` refusal.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new link-swap refusal ("changed location while it was being verified") has no test. The added tests only cover a realpath that throws, a link leading outside the manifest directory, and a link inside it being resolved — none trigger this branch, which is the main new defense against a link swapped mid-verification. Add a test that swaps the directory link between two targets (or mocks realpath to return a different path on the re-check call) and asserts the changed location refusal.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/src/cli/candidate.ts, line 62:

<comment>The new link-swap refusal ("changed location while it was being verified") has no test. The added tests only cover a `realpath` that throws, a link leading outside the manifest directory, and a link inside it being resolved — none trigger this branch, which is the main new defense against a link swapped mid-verification. Add a test that swaps the directory link between two targets (or mocks `realpath` to return a different path on the re-check call) and asserts the `changed location` refusal.</comment>

<file context>
@@ -42,22 +42,31 @@ export async function loadCandidate(manifestPath: string, profile: string): Prom
-    actual = await sha256Of(path);
+    actual = await sha256Of(realFile);
+    // A link swapped while the file was being hashed would make the check above describe some other file.
+    if ((await realpath(path)) !== realFile) return { ok: false, error: `the artifact ${path} changed location while it was being verified` };
   } catch (error) {
     return { ok: false, error: `could not read the artifact ${path}: ${message(error)}` };
</file context>

[Andreas-Froyland]

Fixed in b7143e3. The new test mocks realpath so the artifact resolves to its real location before hashing and somewhere else afterwards, and asserts the 'changed location' refusal and that the re-check ran. I confirmed it fails when the re-check is removed.