From 18ca433852dc8015c01bd316a67abace2459c74c Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 15:08:41 -0500 Subject: [PATCH 1/7] test: re-vendor test-sandbox ce0d966; correct isolation and checkout-write docs (#469) * test: re-vendor test-sandbox ce0d966; correct two doc claims ce0d966 exports the run's actual mode to the command as TEST_SANDBOX_RUN_* (enforcement, isolate, net, pidns) and no longer exports the input knobs, so a nested call cannot mistake an outer run's settings for its own. Nothing in this repo read the old variables. The vendored README said GitHub runners fall back to a direct run; the batch-1 CI run showed them enforcing cgroup+netns. TESTING.md now names the PageShim writes under .tmp/pageshim and .scratch. Assisted-by: AI Signed-off-by: Tim Nunamaker * docs(test-sandbox): say what each missing layer costs; clean the ChatGPT replay bundle Review: the README claimed GitHub runners support full isolation, but only verify + test installs bubblewrap and lifts the AppArmor restriction, so other jobs get cgroup limits without network isolation. The README and TESTING.md now say which layer each host lacks, and TESTING.md lists every place tests write into the checkout. chatgpt-replay.test.mjs left its built bundle under .scratch after a passing run; it now removes it on exit 0. Assisted-by: AI Signed-off-by: Tim Nunamaker * docs(testing): correct which CI jobs use the sandbox and where tests write Assisted-by: AI Signed-off-by: Tim Nunamaker --------- Signed-off-by: Tim Nunamaker --- TESTING.md | 8 +++--- scripts/pageshim/chatgpt-replay.test.mjs | 6 +++- scripts/test-sandbox/README.md | 12 +++++--- scripts/test-sandbox/test-sandbox | 35 ++++++++++++++++++++++-- scripts/test-sandbox/test-sandbox.sha256 | 2 +- 5 files changed, 50 insertions(+), 13 deletions(-) diff --git a/TESTING.md b/TESTING.md index 18956506a..cacfe08e7 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1,6 +1,6 @@ # Testing data-connectors -Every local test command runs through `scripts/test-sandbox/test-sandbox`, a vendored copy of the host-wide test governor (see `scripts/test-sandbox/README.md`). It gives each run a systemd scope under the shared `agent-tests.slice` with 400% CPU, 8 GB and 4096 tasks, one of the host-wide slots, a loopback-only network unless the lane opts out, and a private `/tmp`. Stopping the scope kills every descendant. Each run prints a JSON receipt (wall, CPU, peak memory) on stderr. Where systemd user sessions, bwrap or flock are missing (macOS, GitHub runners), it runs the command directly and says so in the receipt. +Every local test command runs through `scripts/test-sandbox/test-sandbox`, a vendored copy of the host-wide test governor (see `scripts/test-sandbox/README.md`). It gives each run a systemd scope under the shared `agent-tests.slice` with 400% CPU, 8 GB and 4096 tasks, one of the host-wide slots, a loopback-only network unless the lane opts out, and a private `/tmp`. Stopping the scope kills every descendant. Each run prints a JSON receipt (wall, CPU, peak memory) on stderr. Each layer it cannot use on a host is skipped and named in the receipt; see `scripts/test-sandbox/README.md` for what that means on macOS and on GitHub runners. ## Tiers @@ -9,7 +9,7 @@ Every local test command runs through `scripts/test-sandbox/test-sandbox`, a ven | While editing | `npm run test:fast` (root or packages/polyfill-connectors) | Tests related to files changed since `git merge-base HEAD origin/main`, plus uncommitted and untracked files: the static import graph, plus every test in the same `connectors//` directory. Namespace tests run in their own lane. When the selector cannot trust the graph (for example a change under `scripts/related-tests/` or a deleted file), it runs the whole sandboxed lane. | one-connector edit: 42 tests, 4-10 s | | Same, against any base | `npm run test:related -- ` (in packages/polyfill-connectors) | Same selection; on FULL_SUITE it runs both polyfill lanes. | | | Before a PR | `npm test` (root) | Every lane in turn (below). | about 28 min, 16 of them PageShim | -| PR into main | GitHub Actions | `verify + test`, PageShim shards, connector shards, Contract Guardrails, cross-repo integrity. PRs into other branches run no CI. On GitHub runners the sandbox also enforces (cgroup+netns). | `verify + test` Test step 699 s (was 15-18 min at concurrency 2); PageShim shards 588 s and 835 s | +| PR into main | GitHub Actions | `verify + test`, PageShim shards, connector shards, Contract Guardrails, cross-repo integrity. PRs into other branches run no CI. `verify + test` runs its lanes in the sandbox with cgroup limits and network isolation, and Contract Guardrails with cgroup limits only; the PageShim and connector shards and cross-repo integrity run without it (each CI job is its own VM). | `verify + test` Test step 699 s (was 15-18 min at concurrency 2); PageShim shards 588 s and 835 s | The root `npm test` runs these lanes, each in its own sandbox: @@ -25,10 +25,10 @@ The root `npm test` runs these lanes, each in its own sandbox: ## What is shared, per-worker, per-test, locked -- Shared: the checkout, `node_modules`, the Playwright browser cache, the built PageShim bundles of one run. Tests treat them as read-only, except that the namespace tests plant and remove sockets under the repository root (see below). +- Shared: the checkout, `node_modules`, the Playwright browser cache, the built PageShim bundles of one run. Tests write into the checkout in a few places: the namespace tests plant and remove sockets under the repository root (see below); PageShim tests build bundles under `.tmp/pageshim/` and `.scratch/`; `bin/connector-dev.test.ts` creates and removes `packages/polyfill-connectors/runs/` and `connectors//`; `bin/connector-init.test.ts` scaffolds and removes `connectors/zz_init_smoke_/` and `connectors/zz_init_browser_smoke_/` (a killed run leaves them, and the manifest tests then fail); `bin/scenario-cli.test.ts` writes fixture connectors under `packages/polyfill-connectors/tmp/`; and one consent-time-field test makes a temporary directory inside `node_modules`. Do not run two suites in one checkout at once. - Per test file: one `node --test` child process. Module state (PageShim fixture modules with `setLoggedIn`) is per file; tests in one file run one after another. - Per test: temp directories (`mkdtemp` under the sandbox's private `/tmp`), local HTTP servers on port 0, iMessage `chat.db` files, PageShim browsers (one Chromium per `runHarness`, about 140 ms to start). -- Locked (serial): the namespaces lane. Its files share the repository root as global state (`isolation-mechanism.test.ts` plants sockets under it that `scenario-verify-strict.test.ts`'s isolation scan refuses), so one run executes them one at a time. Nothing locks them across two runs in the same checkout, so do not run two suites in one checkout at once. +- Locked (serial): the namespaces lane. Its files share the repository root as global state (`isolation-mechanism.test.ts` plants sockets under it that `scenario-verify-strict.test.ts`'s isolation scan refuses), so one run executes them one at a time. Nothing locks them across two runs in the same checkout. - Never shared: the network. Unit lanes have loopback only; PageShim routes deny every request that no fixture serves. ## Rules diff --git a/scripts/pageshim/chatgpt-replay.test.mjs b/scripts/pageshim/chatgpt-replay.test.mjs index ce7c97de7..49f08d5d6 100644 --- a/scripts/pageshim/chatgpt-replay.test.mjs +++ b/scripts/pageshim/chatgpt-replay.test.mjs @@ -3,7 +3,7 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { mkdirSync } from "node:fs"; +import { mkdirSync, rmSync } from "node:fs"; import { join } from "node:path"; import { buildPageshim } from "./build.mjs"; import { @@ -15,6 +15,10 @@ import { runHarness } from "./harness.mjs"; const out = join(process.cwd(), ".scratch", "pageshim-chatgpt-replay-test"); mkdirSync(out, { recursive: true }); +// Keep a failed run's bundle for debugging; remove it after a passing run. +process.on("exit", (code) => { + if (code === 0) rmSync(out, { recursive: true, force: true }); +}); test("reverse-projects retained ChatGPT records into list pages and details", () => { const projected = projectChatGptExport({ diff --git a/scripts/test-sandbox/README.md b/scripts/test-sandbox/README.md index 061e295a5..de9f83d5d 100644 --- a/scripts/test-sandbox/README.md +++ b/scripts/test-sandbox/README.md @@ -3,11 +3,15 @@ `test-sandbox` runs one test command in a transient systemd scope under the shared `agent-tests.slice`, with CPU, memory and task caps, a host-wide slot limit, loopback-only network by default (`--net host` opts out) and a private `/tmp`. -It prints a JSON receipt on stderr. Where systemd user sessions, bwrap or flock are -missing or unusable (macOS, GitHub runners), it runs the command directly and says so -in the receipt. +It prints a JSON receipt on stderr. Each layer it cannot use is skipped, and the receipt's `enforcement` field +shows what was applied (notes name a missing bwrap only for `--net none` runs): no systemd user session means no cgroup limits, no working bwrap +means no network or /tmp isolation, no flock means no host-wide slots. GitHub's Ubuntu +runners give cgroup limits; they give network isolation only in a job that installs +bubblewrap and lifts the AppArmor user-namespace restriction, which `verify + test` in +polyfill-connectors.yml does. With bash older than 5 (macOS) the command runs directly +and prints one warning line instead of a receipt. Source: `dotfiles/ai/skills/local/test-architecture/scripts/test-sandbox` -Commit: `7ed10f537e7fe5dc370729264ff7bad23a8ae62e` +Commit: `ce0d9665ab830e2c7a4872325f98974484d45fa1` Do not edit here. Update the canonical copy, re-vendor, then update the `.sha256`. diff --git a/scripts/test-sandbox/test-sandbox b/scripts/test-sandbox/test-sandbox index 120c43b04..b2403cfce 100755 --- a/scripts/test-sandbox/test-sandbox +++ b/scripts/test-sandbox/test-sandbox @@ -22,9 +22,27 @@ if [ -z "${BASH_VERSINFO:-}" ] || [ "${BASH_VERSINFO[0]}" -lt 5 ]; then echo "test-sandbox: bash 5 not found; running without enforcement" >&2 # Drop our options (bash 3.2 syntax only): everything after "--" is the command. - for a in "$@"; do shift; [ "$a" = "--" ] && exec "$@"; done + # The scratch contract still holds: TMPDIR is --scratch (created 0700 if + # absent) or a fresh directory removed afterwards. + _prev="" _scr="" _own="" + for a in "$@"; do + shift + if [ "$a" = "--" ]; then + if [ -n "$_scr" ]; then [ -e "$_scr" ] || (umask 077; mkdir -p "$_scr") || exit 2 + else _scr=$(mktemp -d "${TMPDIR:-/tmp}/test-sandbox.XXXXXX") || exit 2; _own=1; fi + TMPDIR=$_scr TMP=$_scr TEMP=$_scr TEST_TMPDIR=$_scr; export TMPDIR TMP TEMP TEST_TMPDIR + "$@"; _rc=$? + [ -n "$_own" ] && rm -rf "$_scr" + exit $_rc + fi + [ "$_prev" = "--scratch" ] && _scr=$a + _prev=$a + done echo "test-sandbox: usage: test-sandbox [options] -- cmd..." >&2; exit 2 fi +# The body is one { ... } block so bash parses all of it before running: +# editing this file while a run is in flight cannot break that run. +{ set -uo pipefail # systemd-run (>= 254) expands ${VAR} in the command line by default, which # mangles shell code and test arguments; turn that off where supported. @@ -78,7 +96,6 @@ have_flock=1; command -v flock >/dev/null || have_flock=0 # A command already inside a sandbox runs directly: a nested call would take a # second slot (and could deadlock with --slots 1) and gain nothing. if [[ -n "${TEST_SANDBOX_ACTIVE:-}" ]]; then exec "$@"; fi -export TEST_SANDBOX_ACTIVE=1 slice=${TEST_SANDBOX_SLICE:-agent-tests} slice=${slice%.slice} slice_unit="$slice.slice" @@ -144,6 +161,12 @@ PY exit 3 fi +# Tell the command (and any nested harness) what it runs under (TEST_SANDBOX_RUN_*), so it can +# verify instead of trust: a nested test-sandbox call runs directly. +# Output names differ from the input knobs (TEST_SANDBOX_NET, _ISOLATE, ...), +# so a sandbox a test starts with TEST_SANDBOX_ACTIVE cleared keeps its defaults. +export TEST_SANDBOX_ACTIVE=1 TEST_SANDBOX_RUN_ENFORCEMENT="$enforcement" TEST_SANDBOX_RUN_ISOLATE="$isolate" TEST_SANDBOX_RUN_PIDNS="$pidns" +if [[ "$enforcement" == *netns* ]]; then export TEST_SANDBOX_RUN_NET=none; else export TEST_SANDBOX_RUN_NET=host; fi runtime=${XDG_RUNTIME_DIR:-$HOME/.cache} state_dir=${TEST_SANDBOX_STATE_DIR:-$runtime/agent-tests} mkdir -p "$state_dir/slots" || die "cannot create slot directory $state_dir/slots" @@ -283,7 +306,11 @@ if [[ $created_scratch -eq 1 ]]; then rm -rf "$scratch"; fi exit "$rc"' if [[ "$disabled" == 1 ]]; then - "$@"; rc=$? + # Same scratch contract as an enforced run. + own_scratch=0 + if [[ -z "$scratch" ]]; then scratch=$(mktemp -d "${TMPDIR:-/tmp}/test-sandbox.XXXXXX") || die "mktemp failed"; own_scratch=1; fi + ( export TMPDIR="$scratch" TMP="$scratch" TEMP="$scratch" TEST_TMPDIR="$scratch"; "$@" ); rc=$? + (( own_scratch )) && rm -rf "$scratch" wall=$(python3 - "$started" <<'PY' import sys,time;print(max(0,time.monotonic()-float(sys.argv[1]))) PY @@ -398,3 +425,5 @@ PY if [[ -n "$receipt" ]]; then printf '%s\n' "$line" >"$receipt" || { echo "test-sandbox: cannot write receipt $receipt" >&2; exit 125; } else printf '%s\n' "$line" >&2; fi exit "$rc" +exit +} diff --git a/scripts/test-sandbox/test-sandbox.sha256 b/scripts/test-sandbox/test-sandbox.sha256 index 75eaf6fc7..55259c461 100644 --- a/scripts/test-sandbox/test-sandbox.sha256 +++ b/scripts/test-sandbox/test-sandbox.sha256 @@ -1 +1 @@ -9f9569a98a0218a38e6ee005c787aa81adec31f23a794a9c3eb4cef8a2090b1d test-sandbox +b51043419454d9836b9a90014fe7a077129711913751bf9c59cba8c747203941 test-sandbox From bd3bce829eae6dfc16428e8c0f53b9eafc91f9b6 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 15:08:45 -0500 Subject: [PATCH 2/7] test: compare only generated sources in the consent-time-field drift check (#472) After any local build (npm run build, and so npm pack or pack-install-run), tsc leaves amazon.generated.d.ts and friends next to each generated slice, and the drift test, which listed the whole directory, failed with every .d.ts as an extra file. The listing now keeps .ts sources and drops .d.ts, so a stale or missing slice still fails and build output does not. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/connector-runtime-consent-time-field.test.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/polyfill-connectors/src/connector-runtime-consent-time-field.test.ts b/packages/polyfill-connectors/src/connector-runtime-consent-time-field.test.ts index dffb342b2..d8080939a 100644 --- a/packages/polyfill-connectors/src/connector-runtime-consent-time-field.test.ts +++ b/packages/polyfill-connectors/src/connector-runtime-consent-time-field.test.ts @@ -121,8 +121,12 @@ test("the generated consent-time-field modules match the manifests on disk", () ); // One slice per manifest, no extra, each current. const expected = readdirSync(join(scratch, "consent-time-fields")).sort(); + // `npm run build` (and so `npm pack`) emits .js and .d.ts next to each + // slice; only the generated sources are the generator's output. assert.deepEqual( - readdirSync(join(tracked, "consent-time-fields")).sort(), + readdirSync(join(tracked, "consent-time-fields")) + .filter((name) => name.endsWith(".ts") && !name.endsWith(".d.ts")) + .sort(), expected, regenerate, ); From e5a47afb3e8ce9c56b92ecb54facc46bac1fa58f Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 16:16:26 -0500 Subject: [PATCH 3/7] test(harness): let high-volume tests skip retaining the connector's raw stdout (#475) Large connector tests hold their output twice, which keeps tens to hundreds of megabytes of extra text in the test process. runConnectorProtocolSubprocess, the test helper that runs a connector as a child process, returns the connector's stdout as raw text (`rawStdout`) and as parsed `messages`. The three largest callers emit many records: apple_health's identity-budget test (500,018 records), whatsapp's 150 MiB .txt test (182,300 messages) and strava's 128 MiB CSV test (104,288 activities). Their raw text alone measured 239 MB, 219 MB and 51 MB. The test process keeps that text for the whole test, although these tests read only the parsed messages. Add a retainRawStdout option, default true. When it is false, the helper never accumulates the raw text, so rawStdout is an empty string. Each chunk of output is still split into lines and parsed into messages as before. The three large tests pass `retainRawStdout: false`. With the option set, they still pass their checks on the connector process's peak memory. Every other caller keeps the default. Tests that read rawStdout must keep the default too, for example the fitbit tests, which search it for planted strings (privacy canaries) that must never reach stdout. The new test runs a small fixture with the option on and off. With it off, the messages match the default run and rawStdout is empty. The default rawStdout holds the same messages as JSON lines. The comparison ignores emitted_at, the timestamp each RECORD message (one connector record) carries when it is emitted, because it differs between any two runs. In a one-off run on the 182,300-message whatsapp fixture, the parsed messages matched between the two modes except two STATE messages (the saved sync cursor). Those two differ in a timestamp and in a list of hashes. Two default runs also differed the same way in a control comparison, so the option does not cause that difference. Not measured: the memory the change frees in the test process. The byte counts are string sizes, not peak memory. Assisted-by: AI Signed-off-by: Tim Nunamaker --- connectors/apple_health/index.test.ts | 6 +++ connectors/strava/no-whole-file-read.test.ts | 2 + .../whatsapp/no-whole-file-read.test.ts | 2 + .../scripts/no-await-in-loops-allowlist.ts | 2 +- .../src/test-harness.test.ts | 38 +++++++++++++++++++ .../polyfill-connectors/src/test-harness.ts | 9 ++++- 6 files changed, 57 insertions(+), 2 deletions(-) diff --git a/connectors/apple_health/index.test.ts b/connectors/apple_health/index.test.ts index 7c59d8af0..42bdcece7 100644 --- a/connectors/apple_health/index.test.ts +++ b/connectors/apple_health/index.test.ts @@ -109,6 +109,7 @@ interface RunOpts { peakRssPollIntervalMs?: number; /** Restrict every health-area stream to specific record ids, as a host may. */ resources?: string[]; + retainRawStdout?: boolean; state?: Record; /** The streams to request. Defaults to every data stream and the receipts. */ streams?: readonly string[]; @@ -155,6 +156,9 @@ async function run(dir: string, opts: RunOpts = {}) { ? {} : { peakRssPollIntervalMs: opts.peakRssPollIntervalMs }), ...(opts.timeoutMs === undefined ? {} : { timeoutMs: opts.timeoutMs }), + ...(opts.retainRawStdout === undefined + ? {} + : { retainRawStdout: opts.retainRawStdout }), start: { ...(opts.fullRefresh ? { collection_mode: "full_refresh" } : {}), scope: { streams: streams.map((name) => scopeFor(name, opts)) }, @@ -3286,8 +3290,10 @@ test("with both identity budgets spent, memory stays within the streaming bound `\n\n \n` + `${parts.join("\n")}\n\n`; await withImportDir({ "export.xml": xml }, async (dir) => { + // Only parsed messages are read here, and there are ~500k of them. const result = await run(dir, { peakRssPollIntervalMs: 200, + retainRawStdout: false, timeoutMs: 540_000, }); assert.ok(result.peakRssBytes !== null, "expected a sampled peak RSS"); diff --git a/connectors/strava/no-whole-file-read.test.ts b/connectors/strava/no-whole-file-read.test.ts index 1ee740471..7feacc47b 100644 --- a/connectors/strava/no-whole-file-read.test.ts +++ b/connectors/strava/no-whole-file-read.test.ts @@ -96,6 +96,8 @@ test("large synthetic activities.csv emits with bounded RSS", async () => { TZ: "UTC", }, peakRssPollIntervalMs: 25, + // Only parsed messages are asserted on, so skip the raw stdout copy. + retainRawStdout: false, start: { scope: { streams: [{ name: "activities" }, ], diff --git a/connectors/whatsapp/no-whole-file-read.test.ts b/connectors/whatsapp/no-whole-file-read.test.ts index a524fdc86..56ce522d1 100644 --- a/connectors/whatsapp/no-whole-file-read.test.ts +++ b/connectors/whatsapp/no-whole-file-read.test.ts @@ -139,6 +139,8 @@ test("a large synthetic WhatsApp .txt export (well past the old 1 GiB cap) parse WHATSAPP_EXPORT_DIR: importRoot, }, peakRssPollIntervalMs: 25, + // Only parsed messages are asserted on, so skip the raw stdout copy. + retainRawStdout: false, start: { scope: { streams: [{ name: "chats" }, { name: "messages" }] }, type: "START", diff --git a/packages/polyfill-connectors/scripts/no-await-in-loops-allowlist.ts b/packages/polyfill-connectors/scripts/no-await-in-loops-allowlist.ts index c0175d67f..0e7b2e0a1 100644 --- a/packages/polyfill-connectors/scripts/no-await-in-loops-allowlist.ts +++ b/packages/polyfill-connectors/scripts/no-await-in-loops-allowlist.ts @@ -686,7 +686,7 @@ export const NO_AWAIT_IN_LOOPS_ALLOWLIST: readonly NoAwaitInLoopsAllowlistEntry[ }, { path: "connectors/apple_health/index.test.ts", - line: 3498, + line: 3504, column: 3, category: "test_assertion_sequencing", note: "small-XML-entry archive cases: each imported in turn by its own subprocess, whose peak RSS is measured alone", diff --git a/packages/polyfill-connectors/src/test-harness.test.ts b/packages/polyfill-connectors/src/test-harness.test.ts index 92bc7de0c..194d7320a 100644 --- a/packages/polyfill-connectors/src/test-harness.test.ts +++ b/packages/polyfill-connectors/src/test-harness.test.ts @@ -482,3 +482,41 @@ test("runConnectorProtocolSubprocess: stream failure evidence forces failed DONE assert.equal(done.error?.code, "stream_collection_failed"); assert.equal(done.error?.retryable, true); }); + +/** `emitted_at` is each run's own clock; blank it so two runs of one fixture compare. */ +function withoutEmittedAt( + messages: readonly EmittedMessage[], +): EmittedMessage[] { + return messages.map((message) => { + if (message.type !== "RECORD") { + return message; + } + return { ...message, emitted_at: "" }; + }); +} + +test("runConnectorProtocolSubprocess: retainRawStdout false keeps the parsed messages and returns no raw text", async () => { + const retained = await runConnectorProtocolSubprocess({ + cwd: PACKAGE_ROOT, + entrypoint: fixturePath("protocol-subprocess-non-browser.ts"), + start: { type: "START", scope: { streams: [{ name: "items" }] } }, + }); + const dropped = await runConnectorProtocolSubprocess({ + cwd: PACKAGE_ROOT, + entrypoint: fixturePath("protocol-subprocess-non-browser.ts"), + retainRawStdout: false, + start: { type: "START", scope: { streams: [{ name: "items" }] } }, + }); + + assert.deepEqual( + withoutEmittedAt(dropped.messages), + withoutEmittedAt(retained.messages), + ); + assert.equal(dropped.rawStdout, ""); + + const rawLines = retained.rawStdout.split("\n").filter((line) => line !== ""); + assert.deepEqual( + rawLines.map((line) => JSON.parse(line)), + retained.messages, + ); +}); diff --git a/packages/polyfill-connectors/src/test-harness.ts b/packages/polyfill-connectors/src/test-harness.ts index 0f1a8f8ea..fbffaf22f 100644 --- a/packages/polyfill-connectors/src/test-harness.ts +++ b/packages/polyfill-connectors/src/test-harness.ts @@ -190,6 +190,10 @@ export interface ConnectorSubprocessOptions { * poll timer is pure overhead for the many tests that don't need a * memory proof. */ peakRssPollIntervalMs?: number; + /** Set false for a run whose stdout is too large to hold twice: `rawStdout` + * is then "" and `messages` is unchanged. Callers that assert on + * `rawStdout` must keep the default (true). */ + retainRawStdout?: boolean; start: { /** Mirrors the real runtime's START `collection_mode` field, so a test can * exercise the owner-triggered full-refresh bypass end to end rather than @@ -273,6 +277,7 @@ export function runConnectorProtocolSubprocess( options: ConnectorSubprocessOptions, ): Promise { const timeoutMs = options.timeoutMs ?? 15_000; + const retainRawStdout = options.retainRawStdout ?? true; return new Promise((resolve, reject) => { const child = spawn( @@ -371,7 +376,9 @@ export function runConnectorProtocolSubprocess( child.stdout?.on("data", (chunk: Buffer) => { const text = chunk.toString(); - rawStdout += text; + if (retainRawStdout) { + rawStdout += text; + } stdoutBuffer += text; let newlineIndex = stdoutBuffer.indexOf("\n"); while (newlineIndex !== -1) { From d21748863f8a8f0496cbda89647ff8914173f590 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 16:50:21 -0500 Subject: [PATCH 4/7] fix(test-harness): decode connector stdout as a stream, not per chunk (#476) runConnectorProtocolSubprocess called chunk.toString() on every pipe chunk, so a multi-byte UTF-8 character split across two chunks became two U+FFFD characters in both rawStdout and the parsed messages. Any test whose connector emits non-ASCII text longer than a pipe buffer could see corrupted records. The child's stdout and stderr now use setEncoding('utf8'), whose decoder carries partial characters across chunks. The new test emits one 300 KB line of 3-byte characters; it failed on the old code with 'no replacement characters' and passes now. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/test-harness.test.ts | 35 +++++++++++++++++++ .../polyfill-connectors/src/test-harness.ts | 11 +++--- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/packages/polyfill-connectors/src/test-harness.test.ts b/packages/polyfill-connectors/src/test-harness.test.ts index 194d7320a..f47d97a51 100644 --- a/packages/polyfill-connectors/src/test-harness.test.ts +++ b/packages/polyfill-connectors/src/test-harness.test.ts @@ -520,3 +520,38 @@ test("runConnectorProtocolSubprocess: retainRawStdout false keeps the parsed mes retained.messages, ); }); + +test("runConnectorProtocolSubprocess: a multi-byte character split across stdout chunks survives", async () => { + const tempDir = await mkdtemp(join(tmpdir(), "connector-subprocess-utf8-")); + const entrypoint = join(tempDir, "utf8-connector.mjs"); + // 3-byte characters, 300 KB in one line: pipe chunks (64 KiB) cannot all + // end on a character boundary, so per-chunk decoding would corrupt some. + const text = "\u20ac".repeat(100_000); + await writeFile( + entrypoint, + [ + `const text = ${JSON.stringify(text)};`, + 'process.stdout.write(JSON.stringify({ type: "RECORD", stream: "items", data: { id: "1", text } }) + "\\n");', + 'process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 1 }) + "\\n");', + ].join("\n"), + ); + try { + const result = await runConnectorProtocolSubprocess({ + cwd: PACKAGE_ROOT, + entrypoint, + start: { type: "START", scope: { streams: [{ name: "items" }] } }, + }); + const record = result.messages.find((m) => m.type === "RECORD") as + | { data: { text: string } } + | undefined; + assert.ok(record, "the RECORD reached messages"); + assert.equal( + record.data.text.includes("\ufffd"), + false, + "no replacement characters", + ); + assert.equal(record.data.text, text); + } finally { + await rm(tempDir, { recursive: true, force: true }); + } +}); diff --git a/packages/polyfill-connectors/src/test-harness.ts b/packages/polyfill-connectors/src/test-harness.ts index fbffaf22f..14e39b83b 100644 --- a/packages/polyfill-connectors/src/test-harness.ts +++ b/packages/polyfill-connectors/src/test-harness.ts @@ -374,8 +374,11 @@ export function runConnectorProtocolSubprocess( ); }, timeoutMs); - child.stdout?.on("data", (chunk: Buffer) => { - const text = chunk.toString(); + // Decode as streams, not per chunk: a multi-byte character split across + // two pipe chunks would otherwise become U+FFFD in both halves. + child.stdout?.setEncoding("utf8"); + child.stderr?.setEncoding("utf8"); + child.stdout?.on("data", (text: string) => { if (retainRawStdout) { rawStdout += text; } @@ -389,8 +392,8 @@ export function runConnectorProtocolSubprocess( } }); - child.stderr?.on("data", (chunk: Buffer) => { - stderr += chunk.toString(); + child.stderr?.on("data", (text: string) => { + stderr += text; }); child.on("error", (err) => { From 184664feaf4d91251d8b1c6dcc4afe2c22a87aea Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 17:14:42 -0500 Subject: [PATCH 5/7] test(harness): sample RSS at start and on output, not only on a timer (#477) Memory-bound tests failed now and then with 'expected a sampled peak RSS': the harness sampled /proc//status only on a 200 ms interval, so a child that finished before the first tick left peakRssBytes null. The harness now also samples when the child starts and on each stdout chunk, so a child that produces records is always sampled while it runs. The new test uses a 60 s interval and a child that exits at once; it failed on the old code and passes now. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../src/test-harness.test.ts | 24 ++++++++++++ .../polyfill-connectors/src/test-harness.ts | 37 +++++++++++-------- 2 files changed, 46 insertions(+), 15 deletions(-) diff --git a/packages/polyfill-connectors/src/test-harness.test.ts b/packages/polyfill-connectors/src/test-harness.test.ts index f47d97a51..f8dd13832 100644 --- a/packages/polyfill-connectors/src/test-harness.test.ts +++ b/packages/polyfill-connectors/src/test-harness.test.ts @@ -555,3 +555,27 @@ test("runConnectorProtocolSubprocess: a multi-byte character split across stdout await rm(tempDir, { recursive: true, force: true }); } }); + +test("runConnectorProtocolSubprocess: a child that exits before the first RSS tick still has a sampled peak", { + skip: process.platform !== "linux", +}, async () => { + const tempDir = await mkdtemp(join(tmpdir(), "connector-subprocess-rss-")); + const entrypoint = join(tempDir, "quick-connector.mjs"); + await writeFile( + entrypoint, + 'process.stdout.write(JSON.stringify({ type: "DONE", status: "succeeded", records_emitted: 0 }) + "\\n");\n', + ); + try { + const result = await runConnectorProtocolSubprocess({ + cwd: PACKAGE_ROOT, + entrypoint, + // Far longer than the child lives, so only the start and output + // samples can produce a value. + peakRssPollIntervalMs: 60_000, + start: { type: "START", scope: { streams: [{ name: "items" }] } }, + }); + assert.notEqual(result.peakRssBytes, null, "expected a sampled peak RSS"); + } finally { + await rm(tempDir, { recursive: true, force: true }); + } +}); diff --git a/packages/polyfill-connectors/src/test-harness.ts b/packages/polyfill-connectors/src/test-harness.ts index 14e39b83b..0baebe8f8 100644 --- a/packages/polyfill-connectors/src/test-harness.ts +++ b/packages/polyfill-connectors/src/test-harness.ts @@ -298,22 +298,28 @@ export function runConnectorProtocolSubprocess( let peakRssBytes: number | null = null; let rssPoller: NodeJS.Timeout | undefined; const { pid } = child; - if (options.peakRssPollIntervalMs && pid) { - rssPoller = setInterval(() => { - try { - const status = readFileSync(`/proc/${pid}/status`, "utf8"); - const match = VM_RSS_STATUS_RE.exec(status); - if (match?.[1]) { - const bytes = Number.parseInt(match[1], 10) * 1024; - peakRssBytes = - peakRssBytes === null ? bytes : Math.max(peakRssBytes, bytes); - } - } catch { - // /proc//status is unavailable (non-Linux, or the process - // already exited between the tick firing and the read) -- leave - // peakRssBytes at its last known value. + const sampleRss = (): void => { + try { + const status = readFileSync(`/proc/${pid}/status`, "utf8"); + const match = VM_RSS_STATUS_RE.exec(status); + if (match?.[1]) { + const bytes = Number.parseInt(match[1], 10) * 1024; + peakRssBytes = + peakRssBytes === null ? bytes : Math.max(peakRssBytes, bytes); } - }, options.peakRssPollIntervalMs); + } catch { + // /proc//status is unavailable (non-Linux, or the process + // already exited between the tick firing and the read) -- leave + // peakRssBytes at its last known value. + } + }; + const sampleRssOnOutput = Boolean(options.peakRssPollIntervalMs && pid); + if (sampleRssOnOutput) { + // The interval alone can miss a child that finishes before its first + // tick; sampling at start and on each stdout chunk (below) always + // yields a value while the child is producing records. + sampleRss(); + rssPoller = setInterval(sampleRss, options.peakRssPollIntervalMs); } const messages: EmittedMessage[] = []; @@ -379,6 +385,7 @@ export function runConnectorProtocolSubprocess( child.stdout?.setEncoding("utf8"); child.stderr?.setEncoding("utf8"); child.stdout?.on("data", (text: string) => { + if (sampleRssOnOutput) sampleRss(); if (retainRawStdout) { rawStdout += text; } From 9bf95f501038d844a620ee5abeb1369aefa92d81 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 17:19:05 -0500 Subject: [PATCH 6/7] fix(scenario,connector-dev): decode connector output as a stream, not per chunk (#480) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit connector-dev, scenario-record and scenario-verify decoded each pipe chunk of a connector's stdout (and the two scenario CLIs its stderr) with chunk.toString(). A multi-byte UTF-8 character split across two chunks became U+FFFD, which a reviewer reproduced as a saved STATE cursor of '€' turning into three replacement characters. Each handler now uses setEncoding('utf8'), as the test harness does since #476. connector-dev's stderr is passed through as bytes and needs no change. Three new tests emit a STATE cursor of 100,001 euro signs, so pipe chunks must split characters, and assert the saved, recorded or verified cursor is exact. All three fail on the old code and pass now. The dot built the patch; the owner moved the scenario fixtures under the package's tmp/ so replay isolation can see them. Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../bin/connector-dev.test.ts | 60 +++++++++ .../polyfill-connectors/bin/connector-dev.ts | 5 +- .../bin/scenario-cli.test.ts | 126 ++++++++++++++++++ .../bin/scenario-record.ts | 10 +- .../bin/scenario-verify.ts | 10 +- 5 files changed, 201 insertions(+), 10 deletions(-) diff --git a/packages/polyfill-connectors/bin/connector-dev.test.ts b/packages/polyfill-connectors/bin/connector-dev.test.ts index 622674483..d5fb1d5f1 100644 --- a/packages/polyfill-connectors/bin/connector-dev.test.ts +++ b/packages/polyfill-connectors/bin/connector-dev.test.ts @@ -23,6 +23,7 @@ import { readFileSync, rmdirSync, rmSync, + writeFileSync, } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -1068,3 +1069,62 @@ test("connector-dev CLI: a connector cannot launder its auth failure into runtim rmSync(tmpDir, { recursive: true, force: true }); } }); + +test("connector-dev CLI: preserves multi-byte STATE cursors across pipe chunks", () => { + const tmpDir = mkdtempSync(join(tmpdir(), "connector-dev-utf8-")); + const connector = `connector-dev-utf8-${String(process.pid)}`; + const entrypoint = join(tmpDir, "connector.mjs"); + const expected = "€".repeat(100_001); + try { + writeFileSync( + entrypoint, + ` +import { once } from "node:events"; +import { setTimeout as delay } from "node:timers/promises"; +await once(process.stdin, "data"); +const emit = (message) => new Promise((resolve, reject) => { + process.stdout.write(JSON.stringify(message) + "\\n", (error) => error ? reject(error) : resolve()); +}); +await emit({ type: "RECORD", stream: "items", key: "utf8", emitted_at: "2026-08-01T00:00:00.000Z", data: { id: "utf8", value: "fixture" } }); +const bytes = Buffer.from(JSON.stringify({ + type: "STATE", stream: "items", cursor: { value: "€".repeat(100_001) } +}) + "\\n"); +// The 64 KiB split lands inside a three-byte character. Waiting after the +// first write makes the split deterministic even when pipe reads coalesce. +const split = 65_536; +await new Promise((resolve, reject) => { + process.stdout.write(bytes.subarray(0, split), (error) => error ? reject(error) : resolve()); +}); +await delay(40); +await new Promise((resolve, reject) => { + process.stdout.write(bytes.subarray(split), (error) => error ? reject(error) : resolve()); +}); +await emit({ type: "DONE", status: "succeeded", records_emitted: 1 }); +process.stdin.resume(); +`, + ); + const result = runCli([ + connector, + "--entrypoint", + entrypoint, + "--summary-out", + join(tmpDir, "summary.json"), + ]); + assert.equal(result.code, 0, result.stderr); + const saved = JSON.parse( + readFileSync(lastStatePathFor(connector), "utf8"), + ) as LastState; + assert.deepEqual(Object.keys(saved.state), ["items"]); + assert.equal( + (saved.state.items as { value: string }).value === expected, + true, + "saved cursor must retain every multi-byte character exactly", + ); + } finally { + rmSync(tmpDir, { recursive: true, force: true }); + rmSync(join(PACKAGE_ROOT, "runs", connector), { + recursive: true, + force: true, + }); + } +}); diff --git a/packages/polyfill-connectors/bin/connector-dev.ts b/packages/polyfill-connectors/bin/connector-dev.ts index 5ab93ab59..52b137af3 100644 --- a/packages/polyfill-connectors/bin/connector-dev.ts +++ b/packages/polyfill-connectors/bin/connector-dev.ts @@ -987,8 +987,9 @@ export function runAndStream( } }; - child.stdout?.on("data", (chunk: Buffer) => { - stdoutBuffer += chunk.toString(); + child.stdout?.setEncoding("utf8"); + child.stdout?.on("data", (chunk: string) => { + stdoutBuffer += chunk; let newlineIndex = stdoutBuffer.indexOf("\n"); while (newlineIndex !== -1) { const line = stdoutBuffer.slice(0, newlineIndex); diff --git a/packages/polyfill-connectors/bin/scenario-cli.test.ts b/packages/polyfill-connectors/bin/scenario-cli.test.ts index a50bbf8f2..190ed5ce5 100644 --- a/packages/polyfill-connectors/bin/scenario-cli.test.ts +++ b/packages/polyfill-connectors/bin/scenario-cli.test.ts @@ -4059,3 +4059,129 @@ test("scenario-verify: replay subprocess forces PLAYWRIGHT_BROWSERS_PATH and PDP rmSync(connectorPath, { force: true }); } }); + +function writeMultiByteConnector(directory: string): string { + const entrypoint = join(directory, "connector.mjs"); + writeFileSync( + entrypoint, + ` +import { once } from "node:events"; +import { setTimeout as delay } from "node:timers/promises"; +await once(process.stdin, "data"); +const emit = (message) => new Promise((resolve, reject) => { + process.stdout.write(JSON.stringify(message) + "\\n", (error) => error ? reject(error) : resolve()); +}); +await emit({ type: "RECORD", stream: "items", key: "utf8", emitted_at: "2026-08-01T00:00:00.000Z", data: { id: "utf8", value: "fixture" } }); +const bytes = Buffer.from(JSON.stringify({ + type: "STATE", stream: "items", cursor: { value: "€".repeat(100_001) } +}) + "\\n"); +// The 64 KiB split lands inside a three-byte character. Waiting after the +// first write makes the split deterministic even when pipe reads coalesce. +const split = 65_536; +await new Promise((resolve, reject) => { + process.stdout.write(bytes.subarray(0, split), (error) => error ? reject(error) : resolve()); +}); +await delay(40); +await new Promise((resolve, reject) => { + process.stdout.write(bytes.subarray(split), (error) => error ? reject(error) : resolve()); +}); +await emit({ type: "DONE", status: "succeeded", records_emitted: 1 }); +process.stdin.resume(); +`, + ); + return entrypoint; +} + +test("scenario-record: preserves multi-byte STATE cursors across pipe chunks", () => { + const tmpDir = mkdtempSync( + join(packageScratchDir(), "scenario-record-utf8-"), + ); + const scenarioPath = join(tmpDir, "utf8.scenario.json"); + try { + const entrypoint = writeMultiByteConnector(tmpDir); + const result = runRecordCli( + [ + "scenario-utf8-fixture", + "--entrypoint", + entrypoint, + "--runs", + "1", + "--out", + scenarioPath, + ], + {}, + ); + assert.equal(result.code, 0, result.stderr); + const scenario = JSON.parse( + readFileSync(scenarioPath, "utf8"), + ) as ConnectorScenario; + assert.equal(scenario.capture.complete, true); + const finalState = scenario.runs[0]?.expected.final_state as + | { items?: { value?: string } } + | undefined; + assert.equal( + finalState?.items?.value === "€".repeat(100_001), + true, + "recorded cursor must retain every multi-byte character exactly", + ); + } finally { + rmSync(tmpDir, { recursive: true, force: true }); + } +}); + +test("scenario-verify: preserves multi-byte STATE cursors across pipe chunks", () => { + const tmpDir = mkdtempSync( + join(packageScratchDir(), "scenario-verify-utf8-"), + ); + const scenarioPath = join(tmpDir, "utf8.scenario.json"); + try { + const entrypoint = writeMultiByteConnector(tmpDir); + // Construct the oracle independently: a broken recorder must not hide + // identical corruption in the verifier by supplying a corrupted cursor. + const scenario: ConnectorScenario = { + format: "pdpp.connector-scenario/1", + connector: { id: "scenario-utf8-fixture" }, + capture: { + captured_at: "2026-08-01T00:00:00.000Z", + evidence_class: "synthetic-spike", + privacy_class: "local-only", + recorder_version: "test", + complete: true, + }, + runs: [ + { + start: { scope: { streams: [{ name: "items" }] }, state: null }, + interactions: [], + expected: { + records: { + items: { + count: 1, + ids: ["utf8"], + ops: ["upsert"], + record_sha256s: [ + hashCanonicalJson({ id: "utf8", value: "fixture" }), + ], + }, + }, + final_state: { items: { value: "€".repeat(100_001) } }, + }, + }, + ], + }; + writeFileSync(scenarioPath, JSON.stringify(scenario)); + const result = runVerifyCli([ + "scenario-utf8-fixture", + "--entrypoint", + entrypoint, + scenarioPath, + ]); + assert.equal( + result.code, + 0, + `stdout=${result.stdout.slice(0, 2000)} stderr=${result.stderr}`, + ); + assert.match(result.stdout, /run 0: PASS/); + } finally { + rmSync(tmpDir, { recursive: true, force: true }); + } +}); diff --git a/packages/polyfill-connectors/bin/scenario-record.ts b/packages/polyfill-connectors/bin/scenario-record.ts index 770c33e78..8b3f10217 100644 --- a/packages/polyfill-connectors/bin/scenario-record.ts +++ b/packages/polyfill-connectors/bin/scenario-record.ts @@ -1684,12 +1684,13 @@ function runRecordSubprocess(args: { } }; - child.stdout.on("data", (chunk: Buffer) => { + child.stdout.setEncoding("utf8"); + child.stdout.on("data", (chunk: string) => { // Activity — resets the inactivity window (a no-op while suspended // for a pending INTERACTION; see `answerInteraction`/`handleParsedLine` // above). watchdog.touch(); - stdoutBuffer += chunk.toString(); + stdoutBuffer += chunk; let newlineIndex = stdoutBuffer.indexOf("\n"); while (newlineIndex !== -1) { const line = stdoutBuffer.slice(0, newlineIndex); @@ -1700,9 +1701,10 @@ function runRecordSubprocess(args: { newlineIndex = stdoutBuffer.indexOf("\n"); } }); - child.stderr.on("data", (chunk: Buffer) => { + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (chunk: string) => { watchdog.touch(); - stderr += chunk.toString(); + stderr += chunk; }); child.on("error", (err) => { watchdog.dispose(); diff --git a/packages/polyfill-connectors/bin/scenario-verify.ts b/packages/polyfill-connectors/bin/scenario-verify.ts index fb51591d0..c827c1808 100644 --- a/packages/polyfill-connectors/bin/scenario-verify.ts +++ b/packages/polyfill-connectors/bin/scenario-verify.ts @@ -1504,7 +1504,8 @@ function runReplaySubprocess(args: { } }; - childStdout.on("data", (chunk: Buffer) => { + childStdout.setEncoding("utf8"); + childStdout.on("data", (chunk: string) => { // Activity — resets the inactivity window (replay's scripted // interaction answering never waits on a human, so this watchdog is // never suspended — see this file's "Inactivity watchdog" section). @@ -1512,7 +1513,7 @@ function runReplaySubprocess(args: { if (hardFailure) { return; } - stdoutBuffer += chunk.toString(); + stdoutBuffer += chunk; let newlineIndex = stdoutBuffer.indexOf("\n"); while (newlineIndex !== -1) { const line = stdoutBuffer.slice(0, newlineIndex); @@ -1529,9 +1530,10 @@ function runReplaySubprocess(args: { newlineIndex = stdoutBuffer.indexOf("\n"); } }); - childStderr.on("data", (chunk: Buffer) => { + childStderr.setEncoding("utf8"); + childStderr.on("data", (chunk: string) => { watchdog.touch(); - stderr += chunk.toString(); + stderr += chunk; }); child.on("error", (err) => { watchdog.dispose(); From 2050697ac1599e7f4b17b804b15ffd60bb55c585 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Sun, 18 Oct 2026 17:54:32 -0500 Subject: [PATCH 7/7] test(google_maps): widen the outcome-proof oracle's hang guard (#482) * test(google_maps): widen the outcome-proof oracle's hang guard The batch-2 CI run failed 'validation oracle timed out after 30000 ms': on a 4-vCPU runner with four test files at once, the streaming pass over the 110 MiB fixture took 38.6 s (11 s alone here). The 30 s limit is a hang guard, not a speed bound, so it is now 90 s, and the test gets its own 240 s timeout so two worst-case passes fit. Assisted-by: AI Signed-off-by: Tim Nunamaker * test(google_maps): state the CI timing in the hang-guard comment accurately Assisted-by: AI Signed-off-by: Tim Nunamaker --------- Signed-off-by: Tim Nunamaker --- connectors/google_maps/validation-from-file.test.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/connectors/google_maps/validation-from-file.test.ts b/connectors/google_maps/validation-from-file.test.ts index be6b12f6c..15614ac17 100644 --- a/connectors/google_maps/validation-from-file.test.ts +++ b/connectors/google_maps/validation-from-file.test.ts @@ -561,7 +561,9 @@ test("validateGoogleMapsTimelineArtifactFromFile: a large sparse export never ca } }); -test("outcome proof: a 110 MiB sparse Timeline export separates streaming from whole-buffer parsing at one heap limit", async () => { +test("outcome proof: a 110 MiB sparse Timeline export separates streaming from whole-buffer parsing at one heap limit", { + timeout: 240_000, +}, async () => { // Deterministic worker-thread oracle, not an in-process memory-delta // measurement (the prior in-process external-memory-delta test here was // proven decorative by mutation testing -- it passed even when reverted @@ -620,7 +622,9 @@ test("outcome proof: a 110 MiB sparse Timeline export separates streaming from w // A worker resource limit reports OOM without aborting the process or // generating a core dump for this expected negative-control outcome. const heapLimitMiB = 128; - const timeoutMs = 30_000; + // A hang guard, not a speed bound: on a 4-vCPU CI runner with four test + // files at once, the streaming pass ran past 30 s (the whole test took 38.6 s). + const timeoutMs = 90_000; type OracleOutcome = { error: (Error & { code?: string }) | undefined; exitCode: number;