Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Stage 0 (proving the assumptions) is under way; the tool itself is not built yet

## Development

Requires Node.js 22.12 or newer.
Requires Node.js 22.18 or newer (the CLI needs no build step because Node strips types directly from that version on).

```sh
npm ci
Expand Down
4 changes: 2 additions & 2 deletions docs/decisions/tool-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ These defaults come from the plan's proposals and the Stage 0 results ([native a
| Area | Decision | Basis |
| --- | --- | --- |
| Repository | `Frogbyte-io/release-qa`, public | Created and made public before Stage 0 |
| Runtime | Node.js 22. `.node-version` pins **22.23.2** exactly; `engines.node` is the floor `>=22.12.0` | The harness ran on Node 22.23.2 (Linux) and 24.13.0 (Windows). CI installs the version in `.node-version`; bump it deliberately |
| Runtime | Node.js 22. `.node-version` pins **22.23.2** exactly; `engines.node` is the floor `>=22.18.0` | The harness ran on Node 22.23.2 (Linux) and 24.13.0 (Windows). 22.18.0 is the first 22.x with type stripping on by default and without an experimental warning (per Node's changelog). CI installs the version in `.node-version`; bump it deliberately |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new engines.node floor >=22.18.0 makes README.md stale: it still states "Requires Node.js 22.12 or newer" (README.md:22). A user following the README could install 22.12–22.17, where type stripping is not on by default and silent, so the no-build load test (which asserts empty stderr) and the CLI would fail. Update the README requirement to match the new floor.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/decisions/tool-layout.md, line 12:

<comment>The new `engines.node` floor `>=22.18.0` makes README.md stale: it still states "Requires Node.js 22.12 or newer" (README.md:22). A user following the README could install 22.12–22.17, where type stripping is not on by default and silent, so the no-build load test (which asserts empty stderr) and the CLI would fail. Update the README requirement to match the new floor.</comment>

<file context>
@@ -9,7 +9,7 @@ These defaults come from the plan's proposals and the Stage 0 results ([native a
 | --- | --- | --- |
 | Repository | `Frogbyte-io/release-qa`, public | Created and made public before Stage 0 |
-| Runtime | Node.js 22. `.node-version` pins **22.23.2** exactly; `engines.node` is the floor `>=22.12.0` | The harness ran on Node 22.23.2 (Linux) and 24.13.0 (Windows). CI installs the version in `.node-version`; bump it deliberately |
+| Runtime | Node.js 22. `.node-version` pins **22.23.2** exactly; `engines.node` is the floor `>=22.18.0` | The harness ran on Node 22.23.2 (Linux) and 24.13.0 (Windows). 22.18.0 is the first 22.x with type stripping on by default and without an experimental warning (per Node's changelog). CI installs the version in `.node-version`; bump it deliberately |
 | Package manager | npm workspaces, one root `package-lock.json` | Plan default; nothing in Stage 0 contradicted it |
 | Language and test | TypeScript 7.0.2, Vitest 5.0.1, `@types/node` 22.20.4, all pinned exactly | Verified together in this change: `npm run typecheck` and `npm test` pass, and a deliberate type error fails the typecheck |
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3c66a24. README now says 22.18 and mentions the no-build guarantee.

| Package manager | npm workspaces, one root `package-lock.json` | Plan default; nothing in Stage 0 contradicted it |
| Language and test | TypeScript 7.0.2, Vitest 5.0.1, `@types/node` 22.20.4, all pinned exactly | Verified together in this change: `npm run typecheck` and `npm test` pass, and a deliberate type error fails the typecheck |
| Native driver (sample apps) | WebdriverIO 9.31.9 `remote()` API + external `tauri-driver` 2.0.6; Edge WebDriver on Windows, `WebKitWebDriver` on Linux | Proven on the sample: 60/60 attempts across three runs per platform |
Expand All @@ -31,7 +31,7 @@ docs/decisions/ decision records

`examples/tauri-smoke` stays outside the workspace on purpose. The packages tested in Stage 0 were built with its own `package-lock.json`, and hoisting its build tooling into a root lockfile would change that resolution without a re-test. Revisit when the sample is consumed by the runner's own tests.

`packages/qa/tsconfig.json` sets `noEmit`. The CLI needs a build step (emit or bundle), which Task 2.2 decides when there is a CLI to build. `bin`, `main` and `exports` are intentionally absent from `package.json` until then: the source is TypeScript that Node cannot load directly, so nothing should be able to import the package before it has a build output.
**No build step (decided in Task 2.2, part 1).** Node 22.18 or newer runs the TypeScript directly by stripping types, so a future CLI entry point can be run with a plain `node` invocation, needing no build step; that entry point does not exist yet (Task 2.2's next part adds `packages/qa/src/cli/`) and this record does not fix its path in advance. What is verified today is `packages/qa/src/index.ts`, which `packages/qa/test/no-build.test.ts` loads under plain Node on every CI runner and asserts that no warning is printed. This only works while the source uses *erasable* syntax (no enums, namespaces or constructor parameter properties), which `erasableSyntaxOnly` in `tsconfig.json` enforces at typecheck time. `packages/qa/tsconfig.json` keeps `noEmit`. This suits running from a checkout; **a published or installed distribution would still need a build or bundle**, and `bin`, `main` and `exports` stay out of `package.json` until that is decided.

## Stage 0 commands and where they go

Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"apps/*"
],
"engines": {
"node": ">=22.12.0"
"node": ">=22.18.0"
},
"scripts": {
"typecheck": "npm run typecheck --workspaces --if-present",
Expand Down
2 changes: 1 addition & 1 deletion packages/qa/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"description": "Release QA runner, contracts, reports and GitHub integration",
"type": "module",
"engines": {
"node": ">=22.12.0"
"node": ">=22.18.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The engines floor was bumped to >=22.18.0 in package.json, but package-lock.json was not regenerated: the root entry (line 15) and the packages/qa entry (line 1519) still record "node": ">=22.12.0". The lockfile is the canonical record of the engine floor, and a subsequent npm install will silently rewrite these entries, so keeping them aligned here avoids stale, misleading metadata and a noisy diff later. Run npm install and commit the regenerated lockfile (both entries should read >=22.18.0).

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/package.json, line 8:

<comment>The engines floor was bumped to >=22.18.0 in package.json, but package-lock.json was not regenerated: the root entry (line 15) and the `packages/qa` entry (line 1519) still record `"node": ">=22.12.0"`. The lockfile is the canonical record of the engine floor, and a subsequent `npm install` will silently rewrite these entries, so keeping them aligned here avoids stale, misleading metadata and a noisy diff later. Run `npm install` and commit the regenerated lockfile (both entries should read `>=22.18.0`).</comment>

<file context>
@@ -5,7 +5,7 @@
   "type": "module",
   "engines": {
-    "node": ">=22.12.0"
+    "node": ">=22.18.0"
   },
   "scripts": {
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3c66a24: ran npm install and verified only the two workspace entries' engines fields changed (root and packages/qa), both now >=22.18.0.

},
"scripts": {
"typecheck": "tsc -p tsconfig.json --noEmit",
Expand Down
88 changes: 88 additions & 0 deletions packages/qa/src/runner/display.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
import { readdir, readFile } from 'node:fs/promises';
import { platform as hostPlatform } from 'node:os';
import { basename } from 'node:path';

/**
* `virtual`: a display server that is not a screen (Xvfb and friends). `real`: the machine's own graphical
* session. `unknown`: something answers on a display but nothing shows which. `none`: no display at all.
*/
export type DisplayKind = 'none' | 'virtual' | 'real' | 'unknown';

/** What the machine says about its graphical session. Gathered separately so classification stays pure. */
export interface DisplayFacts {
platform: NodeJS.Platform;
env: Readonly<Record<string, string | undefined>>;
/** Command lines of the running processes (Linux only; empty elsewhere). */
commandLines: readonly string[];
}

/** X servers that draw to memory or a remote viewer rather than to a screen. */
const VIRTUAL_SERVERS = new Set(['Xvfb', 'Xvnc', 'Xdummy']);

/**
* Never guesses: it names a display virtual only when it can see the virtual server that serves it, and real only
* when the session says it is a desktop session. Anything else that has a display is `unknown`, and the caller
* decides what to do with that. The operating system's name alone never decides.
*/
export function classifyDisplay(facts: DisplayFacts): { kind: DisplayKind; detail: string } {
if (facts.platform === 'win32') {
// Services run in a session named "Services" with no desktop; an interactive one is "Console" or "RDP-Tcp#n".
const session = facts.env.SESSIONNAME ?? '';
return session !== '' && session !== 'Services' ? { kind: 'real', detail: `interactive session ${session}` } : { kind: 'none', detail: 'no interactive session' };
}
if (facts.platform !== 'linux') {
// DISPLAY/WAYLAND_DISPLAY are not required by this platform's own native GUI apps, so their presence is only
// weak evidence and their absence is not proof either way — except that with no evidence at all, a headless
// host must not be handed a display capability it cannot back up.
const { DISPLAY: otherDisplay, WAYLAND_DISPLAY: otherWayland } = facts.env;
if (!otherDisplay && !otherWayland) return { kind: 'none', detail: `no display evidence on ${facts.platform}` };
return { kind: 'unknown', detail: `cannot tell on ${facts.platform}` };
}

const { DISPLAY: display, WAYLAND_DISPLAY: wayland, XDG_SESSION_TYPE: session } = facts.env;
if (!display && !wayland) return { kind: 'none', detail: 'neither DISPLAY nor WAYLAND_DISPLAY is set' };

if (display) {
const target = displayNumber(display);
for (const line of facts.commandLines) {
const [program = '', ...args] = line.trim().split(/\s+/);
const name = basename(program);
if (VIRTUAL_SERVERS.has(name) && target !== undefined && args.some((arg) => displayNumber(arg) === target)) return { kind: 'virtual', detail: `${name} serving ${display}` };
}
}
if ((session === 'x11' && display) || (session === 'wayland' && (wayland || display))) return { kind: 'real', detail: `${session} desktop session` };
return { kind: 'unknown', detail: `a display is set (${display ?? wayland}) but it is not evidently a desktop session or a virtual server` };
}

/**
* The `:N` display number a DISPLAY-like string names, ignoring any host prefix and any `.screen` suffix: Xvfb's own
* argument never carries a screen number (screens are configured separately, with `-screen`), but a client's
* DISPLAY may still name one explicitly, and ":99.0" is the same endpoint as ":99".
*/
function displayNumber(raw: string): string | undefined {
const match = /:(\d+)(?:\.\d+)?$/.exec(raw);
return match === null ? undefined : `:${match[1]}`;
}

/** Reads what this machine says. Best effort: anything unreadable is simply absent from the facts. */
export async function readDisplayFacts(): Promise<DisplayFacts> {
const platform = hostPlatform();
return { platform, env: process.env, commandLines: platform === 'linux' ? await linuxCommandLines() : [] };
}

// Every process is read, not just the first few thousand: missing the one X server that happens to be serving this
// display would misclassify a virtual display as unknown or real. Read in bounded batches rather than opening every
// /proc/<pid>/cmdline at once, so a host with many thousands of processes cannot exhaust file descriptors.
const CMDLINE_BATCH_SIZE = 256;

async function linuxCommandLines(): Promise<string[]> {
const entries = await readdir('/proc').catch(() => [] as string[]);
const pids = entries.filter((name) => /^\d+$/.test(name));
const lines: string[] = [];
for (let start = 0; start < pids.length; start += CMDLINE_BATCH_SIZE) {
const batch = pids.slice(start, start + CMDLINE_BATCH_SIZE);
const read = await Promise.all(batch.map((pid) => readFile(`/proc/${pid}/cmdline`, 'utf8').then((text) => text.split('\0').join(' ').trim(), () => '')));
lines.push(...read);
}
return lines.filter((line) => line !== '');
}
42 changes: 29 additions & 13 deletions packages/qa/src/runner/environment.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,27 @@ import { readFile } from 'node:fs/promises';
import { arch, platform, release } from 'node:os';
import type { EnvironmentProfile } from '../model/project.ts';
import type { MeasuredEnvironment } from '../model/result.ts';
import { classifyDisplay, readDisplayFacts, type DisplayKind } from './display.ts';

/** How the runner asks the machine what it can do. Injectable so tests need no display or sound card. */
export interface EnvironmentProbes {
display(): Promise<boolean>;
audio(): Promise<boolean>;
/** What kind of display it is, when the probe can tell. Without it a display is of unknown kind. */
describeDisplay?(): Promise<{ kind: DisplayKind; detail: string }>;
}

/**
* Heuristics, not proof. `display` says a graphical session appears to be reachable; `audio` says the sound
* subsystem appears to be running. Neither checks that a particular device exists or that a virtual display is
* not standing in for a real one; a scenario that needs that must check it itself.
* Heuristics, not proof. `display` says a graphical session appears to be reachable and `describeDisplay` says
* whether it looks virtual or real, or admits it cannot tell; `audio` says the sound subsystem appears to be
* running. Neither checks that a particular device exists.
*/
export const defaultProbes: EnvironmentProbes = {
async display() {
if (platform() === 'win32') {
// Services run in a non-interactive session, which is named "Services".
const session = process.env.SESSIONNAME;
return session !== undefined && session !== '' && session !== 'Services';
}
return Boolean(process.env.DISPLAY || process.env.WAYLAND_DISPLAY);
return classifyDisplay(await readDisplayFacts()).kind !== 'none';
},
async describeDisplay() {
return classifyDisplay(await readDisplayFacts());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: On macOS without DISPLAY or WAYLAND_DISPLAY, this default descriptor reports an unknown display, so inspectEnvironment incorrectly grants the display capability and can run GUI scenarios on a headless host. Preserve none when no display variable is present before applying the non-Linux fallback, or make classifyDisplay return none for that case.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/qa/src/runner/environment.ts, line 26:

<comment>On macOS without `DISPLAY` or `WAYLAND_DISPLAY`, this default descriptor reports an `unknown` display, so `inspectEnvironment` incorrectly grants the `display` capability and can run GUI scenarios on a headless host. Preserve `none` when no display variable is present before applying the non-Linux fallback, or make `classifyDisplay` return `none` for that case.</comment>

<file context>
@@ -3,26 +3,27 @@ import { readFile } from 'node:fs/promises';
+    return classifyDisplay(await readDisplayFacts()).kind !== 'none';
+  },
+  async describeDisplay() {
+    return classifyDisplay(await readDisplayFacts());
   },
   async audio() {
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3c66a24. On any platform that's neither Linux nor Windows, no DISPLAY/WAYLAND_DISPLAY at all is now 'none'; either being set (but unclassifiable further) is still 'unknown'. Test: 'no display evidence at all is reported as none, not guessed as a working display'.

},
async audio() {
if (platform() === 'win32') return (await run('sc', ['query', 'Audiosrv'])).includes('RUNNING');
Expand All @@ -39,31 +40,46 @@ function run(command: string, args: readonly string[]): Promise<string> {

export interface InspectedEnvironment {
environment: MeasuredEnvironment;
/** What the machine's display is, for reports and `doctor`. */
display: { kind: DisplayKind; detail: string };
/** Set when the machine is not what the profile asks for (operating system or architecture). */
profileMismatch?: string;
}

const OS_NAMES: Record<string, string> = { win32: 'windows', linux: 'linux', darwin: 'macos' };
const ARCH_NAMES: Record<string, string> = { x64: 'x86_64', arm64: 'aarch64' };

/** Measures this machine. A probe that throws means the capability is absent, never a crash of the run. */
/**
* Measures this machine. A probe that throws means the capability is absent, never a crash of the run. A display
* is a `display` capability, and only a display that is evidently a desktop session is also a `real-display`.
*/
export async function inspectEnvironment(profile: EnvironmentProfile, probes: EnvironmentProbes = defaultProbes, toolVersion = '0.0.0'): Promise<InspectedEnvironment> {
// Run inside a promise so a probe that throws before returning one is caught too, not only one that rejects.
const has = (probe: () => Promise<boolean>): Promise<boolean> => Promise.resolve().then(probe).catch(() => false);
const [display, audio] = await Promise.all([has(() => probes.display()), has(() => probes.audio())]);
const [audio, display] = await Promise.all([has(() => probes.audio()), describeDisplay(probes)]);

const os = OS_NAMES[platform()] ?? platform();
const architecture = ARCH_NAMES[arch()] ?? arch();
const environment: MeasuredEnvironment = {
os,
osVersion: release(),
arch: architecture,
capabilities: [...(audio ? ['audio'] : []), ...(display ? ['display'] : [])],
capabilities: [...(audio ? ['audio'] : []), ...(display.kind !== 'none' ? ['display'] : []), ...(display.kind === 'real' ? ['real-display'] : [])],
toolVersion,
};

const problems: string[] = [];
if (os !== profile.os) problems.push(`this machine is ${os}, the profile expects ${profile.os}`);
if (architecture !== profile.arch) problems.push(`this machine is ${architecture}, the profile expects ${profile.arch}`);
return problems.length === 0 ? { environment } : { environment, profileMismatch: problems.join('; ') };
return problems.length === 0 ? { environment, display } : { environment, display, profileMismatch: problems.join('; ') };
}

/** A probe that can only say yes or no describes a display of unknown kind; one that throws describes none. */
async function describeDisplay(probes: EnvironmentProbes): Promise<{ kind: DisplayKind; detail: string }> {
try {
if (probes.describeDisplay !== undefined) return await probes.describeDisplay();
return (await probes.display()) ? { kind: 'unknown', detail: 'a display is present but its kind cannot be told' } : { kind: 'none', detail: 'no display' };
} catch {
return { kind: 'none', detail: 'the display probe failed' };
}
}
21 changes: 21 additions & 0 deletions packages/qa/test/no-build.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { execFile } from 'node:child_process';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { expect, test } from 'vitest';

const run = promisify(execFile);
const packageDir = join(dirname(fileURLToPath(import.meta.url)), '..');

// The CLI has no build step: Node (22.18 or newer) runs the TypeScript directly by stripping types. That only works
// while the source uses erasable syntax alone (no enums, namespaces or constructor parameter properties), which the
// `erasableSyntaxOnly` compiler option enforces. The floor of 22.18 is when stripping became default and silent.
test('the package loads under Node type stripping, so the CLI needs no build, and it prints no warning', async () => {
const { stdout, stderr } = await run(
process.execPath,
['-e', "import('./src/index.ts').then((m) => console.log(typeof m.executeScenario + ' ' + typeof m.evaluate + ' ' + typeof m.renderReport))"],
{ cwd: packageDir },
);
expect(stdout.trim()).toBe('function function function');
expect(stderr).toBe('');
});
Loading
Loading