Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ The first targets are Tauri applications on Windows and Linux, with Dot X as the
## Status

Stage 0 (proving the assumptions) is complete. The runner (environment checks, scenario execution, the durable run
journal) and the CLI's local commands (`doctor`, `designate`, `status`, `reset`, `run`, `resume`) exist. The Tauri
driver adapter and a runnable sample consumer, the dashboard and the GitHub integration do not exist yet.
journal), the CLI's local commands (`doctor`, `designate`, `status`, `reset`, `run`, `resume`), the Tauri driver adapter
and a runnable sample consumer exist: the sample passes through `release-qa run` on Windows and on Ubuntu 24.04 with
Xvfb ([guide](docs/guides/run-the-sample.md), [evidence](examples/tauri-smoke/qa/evidence/)). The dashboard and the
GitHub integration do not exist yet.

- [Native automation](docs/decisions/native-automation.md): unchanged packaged Tauri apps can be driven on Windows and Ubuntu. The Dot X feasibility check is still open.
- [GitHub merge gate](docs/decisions/github-gate.md): a no-service required check works, with documented design changes and unproven items.
Expand All @@ -33,7 +35,8 @@ node packages/qa/src/cli/main.ts run --project <qa/project.json> --candidate <ca
node packages/qa/src/cli/main.ts resume --run <run id> [--state <dir>] [--json]
```

`doctor` and `run` need a consumer's `qa/project.json`; this repository does not ship a runnable sample consumer yet.
`doctor` and `run` need a consumer's `qa/project.json`; the sample's is [`examples/tauri-smoke/qa`](examples/tauri-smoke/qa),
and [the setup guide](docs/guides/run-the-sample.md) takes a fresh machine to a passing run.
`run` also needs a [local candidate manifest](docs/decisions/local-runs.md#the-local-candidate-manifest) naming the
file to test and its SHA-256, which is checked before anything is installed.

Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/local-runs.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Decision: running a suite from a checkout (Task 2.2, part 3a)

Status: **implemented** for the CLI's `run`, `resume` and `reset`, against fixture consumer projects. Running the real sample (`examples/tauri-smoke`) through a Tauri driver adapter is part 3b.
Status: **implemented** for the CLI's `run`, `resume` and `reset`: tested against fixture consumer projects, and run for real with the sample (`examples/tauri-smoke`) through the Tauri driver adapter on Windows and Ubuntu 24.04 ([guide](../guides/run-the-sample.md), [evidence](../../examples/tauri-smoke/qa/evidence/)).

## The local candidate manifest

Expand Down
17 changes: 12 additions & 5 deletions docs/decisions/tool-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,23 @@ docs/decisions/ decision records

## Stage 0 commands and where they go

The Stage 0 commands are not turned into npm scripts yet, because they need a designated machine, installed drivers and a built package rather than just `npm ci`. They stay documented and runnable in [`experiments/native-automation`](../../experiments/native-automation) until Stage 2 promotes them.
The native-automation harness is promoted (Task 2.2): the same check now runs through `release-qa run` with the sample's
`qa/` consumer, following [the setup guide](../guides/run-the-sample.md), with [evidence](../../examples/tauri-smoke/qa/evidence/)
from Windows and a fresh Ubuntu 24.04 machine. It still needs a designated machine, installed drivers and a built package,
so it is not an npm script and not in CI. The Stage 0 harness stays in [`experiments/native-automation`](../../experiments/native-automation) as the record of Stage 0.

| Stage 0 command | Promoted to |
| --- | --- |
| `run-attempts.mjs` (launch, save, read disk, restart, clear) | `packages/qa/src/drivers/tauri.ts` and the runner (Task 2.1/2.2), driven by `release-qa run` |
| Driver/WebView2 version match, display presence, no already-running instance | `release-qa doctor` |
| Hash the installed file, not the build tree | Candidate preparation (Task 3.1) and the runner's install step |
| `run-attempts.mjs` (launch, save, read disk, restart, clear) | `packages/qa/src/drivers/tauri.ts`, the runner and `examples/tauri-smoke/qa/`, driven by `release-qa run` |
| Display presence | `release-qa doctor` |
| No already-running instance, native driver present, driver ports free | the Tauri adapter, before it starts anything |
| Driver/WebView2 version match | a setup step in the guide; not checked by the tool |
| Hash the installed file, not the build tree | Candidate preparation (Task 3.1). The runner verifies the candidate package; the sample's installed binary was checked by hand (see the evidence) |
| `experiments/github-gate/*.sh`, `sandbox/scripts/qa-evaluate.mjs` | `packages/qa/src/github/*` and consumer workflows (Tasks 3.1-3.3), applying the changes listed in [github-gate.md](github-gate.md) |

The repeatable smoke check for now is `experiments/native-automation/run-attempts.mjs` itself, plus `experiments/github-gate/run-core.sh`. Both are clearly separate from the shipped package: nothing under `packages/` or `apps/` imports from `experiments/` or `examples/`.
The repeatable smoke check is now the sample's `release-qa run` (see the guide), plus `experiments/github-gate/run-core.sh`. Nothing under `packages/` or `apps/` imports from `experiments/` or `examples/`; the sample's `qa/` files import the package (`packages/qa/src/...`) by relative path, since nothing is published.

**WebdriverIO dependency (Task 2.2).** `packages/qa` depends on `webdriverio` 9.31.9, pinned exactly as proven in Stage 0, and only `packages/qa/src/drivers/tauri.ts` imports it, so the package's main entry point does not load it. `npm audit` reports one advisory through it: `extract-zip` ≤2.0.1 ([GHSA-jmr9-qjv8-65gv](https://github.com/advisories/GHSA-jmr9-qjv8-65gv), [GHSA-7pqw-9j4j-h8q3](https://github.com/advisories/GHSA-7pqw-9j4j-h8q3)), symlink path traversal when extracting an archive, reached through `@puppeteer/browsers`, which WebdriverIO uses to download browsers and drivers. The adapter never takes that path: it connects to an already-running `tauri-driver` by host and port, and drivers are fetched separately and hash-checked (see the guide). No patched `extract-zip` exists; npm's only offered fix is WebdriverIO below 8.15. Revisit when WebdriverIO or `@puppeteer/browsers` moves off `extract-zip`.

## CI

Expand Down
105 changes: 105 additions & 0 deletions docs/guides/run-the-sample.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Running the sample through Release QA

This guide takes a fresh machine to a passing `release-qa run` of the sample app's persistence scenario
([`examples/tauri-smoke`](../../examples/tauri-smoke)): the packaged app is installed, saved to, checked on disk,
restarted, cleared, restarted again and removed, all driven through WebDriver as proven in Stage 0
([decision](../decisions/native-automation.md)).

It needs a **designated machine**: an interactive Windows desktop session, or Linux with a display (Xvfb is enough
for this sample). Hosted CI runners are not used for this; see [tool layout](../decisions/tool-layout.md#ci).

The same four pieces are needed everywhere, pinned to what Stage 0 proved:

| Piece | Version | Why pinned |
| --- | --- | --- |
| Node.js | 22.18 or newer (`.node-version` has the exact CI version) | runs the CLI with no build step |
| `tauri-driver` | 2.0.6, `cargo install --locked` | the driver chain Stage 0 proved |
| Native driver | Windows: Microsoft Edge WebDriver **matching the installed WebView2 runtime**; Linux: `WebKitWebDriver` from `webkit2gtk-driver` | a mismatched Edge WebDriver refuses to start sessions |
| The sample's package | built on the same machine type: NSIS installer (Windows), `.deb` (Linux) | the candidate is the file you built |

## Windows 11

Prerequisites: Git, Node.js, [Rust](https://rustup.rs), and the [Tauri 2 prerequisites](https://v2.tauri.app/start/prerequisites/)
(Microsoft C++ Build Tools; WebView2 is part of Windows 11). In PowerShell, from a clone of this repository:

```powershell
cargo install tauri-driver --locked --version 2.0.6

# The Edge WebDriver must match the WebView2 runtime exactly. Read its version, then fetch that driver.
# A machine-wide WebView2 registers under HKLM, a per-user one under HKCU; the first one set is the installed runtime.
$v = 'HKLM:\SOFTWARE\WOW6432Node\Microsoft\EdgeUpdate\Clients\{F3017226-FE2A-4295-8BDF-00C3A9A7E4C5}',
'HKCU:\SOFTWARE\Microsoft\EdgeUpdate\Clients\{F3017226-FE2A-4295-8BDF-00C3A9A7E4C5}' |
ForEach-Object { (Get-ItemProperty $_ -ErrorAction SilentlyContinue).pv } |
Where-Object { $_ -and $_ -ne '0.0.0.0' } | Select-Object -First 1
if (-not $v) { throw 'WebView2 Runtime is not installed' }
$tools = "$env:LOCALAPPDATA\release-qa\tools\msedgedriver-$v"
New-Item -ItemType Directory -Force $tools | Out-Null
Invoke-WebRequest "https://msedgedriver.microsoft.com/$v/edgedriver_win64.zip" -OutFile "$tools\edgedriver_win64.zip"
Expand-Archive "$tools\edgedriver_win64.zip" -DestinationPath $tools -Force
Get-FileHash "$tools\msedgedriver.exe" -Algorithm SHA256 # record it with your run
$env:RELEASE_QA_NATIVE_DRIVER = "$tools\msedgedriver.exe"

npm ci
cd examples\tauri-smoke
npm ci
npm run build:windows
$candidate = node scripts\write-candidate.mjs
cd ..\..

node packages\qa\src\cli\main.ts designate
node packages\qa\src\cli\main.ts doctor --project examples\tauri-smoke\qa\project.json --profile windows
node packages\qa\src\cli\main.ts run --project examples\tauri-smoke\qa\project.json --candidate $candidate --profile windows --suite release
```

The app window opens and closes three times. Leave the machine alone while it runs.

The sample must not already be installed for your user: the run refuses, rather than taking over an installation it did
not make. The installer goes into the test root (`/S /D=<root>\smoke-app`); outside it, it adds an uninstall entry,
Start Menu and Desktop shortcuts, and `HKCU\Software\frogbyte\Release QA Smoke`. Cleanup runs the uninstaller in place
and removes that key, which the uninstaller leaves behind.

## Ubuntu 24.04

A fresh machine, a VM or a WSL2 distribution all work; the sample only needs a virtual display. From a clone of this
repository:

```sh
sudo apt-get update
sudo apt-get install -y build-essential curl wget file pkg-config libwebkit2gtk-4.1-dev libxdo-dev libssl-dev \
libayatana-appindicator3-dev librsvg2-dev xvfb webkit2gtk-driver git
# Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && . "$HOME/.cargo/env"
cargo install tauri-driver --locked --version 2.0.6
# Node.js: any 22.18+ install; for example the official build pinned in .node-version
```

Then:

```sh
npm ci
(cd examples/tauri-smoke && npm ci && npm run build:linux)
candidate=$(node examples/tauri-smoke/scripts/write-candidate.mjs)

node packages/qa/src/cli/main.ts designate
xvfb-run -a -s "-screen 0 1280x1024x24" node packages/qa/src/cli/main.ts doctor --project examples/tauri-smoke/qa/project.json --profile linux
xvfb-run -a -s "-screen 0 1280x1024x24" node packages/qa/src/cli/main.ts run --project examples/tauri-smoke/qa/project.json --candidate "$candidate" --profile linux --suite release
```

On **WSL2**, WSLg sets `WAYLAND_DISPLAY`, and GTK then draws through WSLg instead of the virtual X display; run
`unset WAYLAND_DISPLAY` first so the run really uses Xvfb (a headless Ubuntu machine has no `WAYLAND_DISPLAY` to begin
with). `doctor` shows which display the run sees.

`RELEASE_QA_NATIVE_DRIVER` defaults to `/usr/bin/WebKitWebDriver` on Linux. The `.deb` is **unpacked** into the test
root with `dpkg-deb -x` rather than installed with `apt`: the app binary is the packaged one byte for byte, no root
access is needed, and nothing is installed system-wide, but the package manager's own steps and desktop integration are
not exercised.

## What a run leaves

- A passing run exits `0` and prints `linux/persistence: passed` (or `windows/...`). The run's journal and summary are
under `.release-qa/runs/<run id>/`.
- The app, its data directory (`%APPDATA%\dev.frogbyte.releaseqa.smoke` or `~/.local/share/dev.frogbyte.releaseqa.smoke`)
and everything the installer created are removed. The sample's data directory is wiped at the start and end of
every run, so do not use the sample app for anything else on that machine.
- If a run is interrupted, `resume --run <id>` continues it; if cleanup could not finish, `reset` clears what it left
once you have fixed the cause. See [local runs](../decisions/local-runs.md).
Loading
Loading