diff --git a/.xum/skills/verify-xum/SKILL.md b/.xum/skills/verify-xum/SKILL.md new file mode 100644 index 00000000000..ecb28936a2e --- /dev/null +++ b/.xum/skills/verify-xum/SKILL.md @@ -0,0 +1,230 @@ +--- +name: verify-xum +description: Drives the real Xum web UI (a `make dev-server-sandbox` instance with the mock AI, isolated XUM_ROOT and free ports) through agent-browser to prove a change works the way a user sees it, with screenshots, ARIA snapshots and on-disk side effects as evidence. Use to verify a UI or backend change in this checkout end to end, to reproduce a reported Xum bug in the browser, or before claiming a Xum feature works. +--- + +# Verify Xum in the browser + +This skill starts one disposable Xum dev server from this checkout, drives its web UI with +`agent-browser`, and keeps the evidence after teardown. The mock AI (`XUM_MOCK_AI=1`) answers every +chat turn, so a run makes no model calls and costs nothing. + +The feature map in [features/README.md](features/README.md) has one recipe per user-facing feature. +Pick the recipe for the behavior you changed and drive every entry point it lists. + +Surfaces: + +- Primary: the web UI that `make dev-server` serves (Vite on a free port, backend `xum server` on + another). Source edits hot-reload into the running instance. +- Electron: the same renderer in the desktop shell. This skill does not launch or prove it. See + the [dev-desktop-sandbox](../dev-desktop-sandbox/SKILL.md) and [electron](../electron/SKILL.md) + skills. +- CLI: `xum api` against the sandbox (see Drive). Headless `xum run` needs no server. Run it + under `env -i` with its own two roots under the run's scratch folder, which cleanup deletes. It + refuses to start without a provider key, so pass a fake one: the mock answers, and your real + keys stay out. + + ```bash + env -i PATH="$PATH" HOME="$HOME" TMPDIR="$VERIFY_XUM_RUN/tmp" XUM_MOCK_AI=1 XUM_DISABLE_TELEMETRY=1 \ + ANTHROPIC_API_KEY=sk-ant-verify-xum-fake \ + XUM_ROOT="$VERIFY_XUM_RUN/scratch/cli-root" \ + XUM_RUN_SESSION_ROOT="$VERIFY_XUM_RUN/scratch/cli-sessions" \ + bun src/cli/index.ts run --dir "$VERIFY_XUM_PROJECT" --json "[mock:list-languages] hi" + ``` + +- Model-driven explorers and story tests run only in the bug-bash sandbox. See the + [bug-bash](../bug-bash/SKILL.md) skill. + +## Launch + +The script runs on Linux only: it needs `setsid`, `flock`, `ss` and `/proc`. Bun on PATH must be +the version in `package.json` `packageManager`. Launch refuses another Bun. + +```bash +.xum/skills/verify-xum/scripts/verify-xum.sh launch +``` + +Launch runs `make dev-server-sandbox` with `--clean-providers --clean-projects`, the mock AI and an +`env -i` allowlist, in its own process group. It returns when the app is usable, in about 15 s when +`dist/` is already built. It gives up when the whole launch, seed included, takes longer than +`VERIFY_XUM_READY_TIMEOUT` seconds (default 300). +Ready means all of these hold: + +- the backend that nodemon restarts after the first watch build owns `server.lock` and answers + `/health`, +- Vite serves the app, +- the seed is done: onboarding marked seen, a fake Anthropic provider on a dead loopback port (the + composer refuses to send without a provider), the trusted project `demo-app` (a throwaway git + repo) and the workspace `Verify playground` with its worktree set up. + +It prints the run's `export` lines and the URL of the seeded workspace. A failed launch prints the +server log tail and cleans up. Launch refuses while the process group of this checkout's newest +run is alive, by the same identity check as cleanup: two dev servers in one checkout share `dist/` +and `node_modules/.vite`. The group's first process writes the group id into the run's +`state.env` before it starts the server. From that moment, right after the spawn, the check and +cleanup find the group even when the launcher was killed. Until then the group holds the launch +lock, so a second launch waits for it. Launch also stops at once when the +sandbox copied a `config.json` (the recipes assume default preferences) and when the backend +crashes after its rebuild (nodemon then waits for a file change that never comes). + +## Doctor + +Run it first whenever anything looks off. It changes nothing. + +```bash +.xum/skills/verify-xum/scripts/verify-xum.sh doctor +``` + +Every check line must read `ok`. A `note` line only informs. It checks that the process group +lives and carries the run's id (`VERIFY_XUM_SERVER_OF` in its env), the checkout's newest-run link +names the run, the backend pid in the sandbox `server.lock` and the app port belong to the group, +`/health` answers, `/version` reports the launch commit, and the backend env has `XUM_MOCK_AI=1`. +The backend always has an auth token: launch runs make under `env -i`, which drops +`XUM_DEV_SERVER_NO_AUTH`. (A checkout from before #7904 starts it without one, and the skill works +the same.) Doctor and launch read the token from `server.lock` and send it to curl on stdin, never +in argv. +Exit 1 means do not drive this instance: run cleanup, then launch again. + +## Drive + +Every bash call starts by loading the run. This sets `AGENT_BROWSER_SESSION` (a session name only +this run uses) and the `VERIFY_XUM_*` paths and URLs. It also unsets the Git variables that point +git at another repository (`GIT_DIR`, `GIT_INDEX_FILE` and the rest of +`git rev-parse --local-env-vars`), so the recipes' git commands act on the demo repo: + +```bash +eval "$(.xum/skills/verify-xum/scripts/verify-xum.sh env)" +.xum/skills/verify-xum/scripts/verify-xum.sh open "/workspace/$VERIFY_XUM_WORKSPACE_ID" +.xum/skills/verify-xum/scripts/verify-xum.sh idle +agent-browser snapshot -i +agent-browser find role button click --name "Open settings" --exact +``` + +- Prefer `find role --name "" --exact` and the names in the + feature map over `@eN` refs, which change with every snapshot. Without `--exact`, `--name` + matches substrings and can act on a different element. +- Open pages with `verify-xum.sh open `, not `agent-browser open`. `make dev-server` starts + the backend with an auth token, and the first page of a browser session needs it as + `?token=` (the app keeps it and strips it from the address bar). `open` builds that URL in + node from `server.lock` and hands it to `agent-browser batch` on stdin, so the token is in no + command line or environment. CAUTION: never put the token in an `agent-browser` argument. Every + local account can read a process's command line while it runs. +- CAUTION: Run this skill only on a host where you are the only local account. agent-browser + starts Chrome with a DevTools port on `127.0.0.1` (`--remote-debugging-port=0`), which asks for + no credentials and which agent-browser cannot turn off. While the signed-in app is open, any + local account that finds that port can control the browser, read the page and take the stored + token. +- `open` turns off agent-browser's live stream for the session first, every time, and refuses to + open the page while it is on. agent-browser 0.33 serves each session's screen and input as a + WebSocket on `127.0.0.1` that any local account can connect to, starts it with each new + daemon, and has no setting to keep it off. CAUTION: never run `agent-browser stream enable` in + a run's session. Another account could then watch and drive the signed-in app. +- `idle` waits until the page settles: no CSS transition runs, and on a `/workspace/` page the + `Conversation transcript` log is neither loading nor streaming (`aria-busy="false"`). A click + while the right sidebar slides open lands where the target was and is lost. `idle` exits 3 + when the page shows ` failed to load.` (see Gotchas). +- `send-ready` waits until the composer's `Send message` button is enabled. Fill the message, run + `send-ready`, then send. The workspace composer keeps Send disabled until the transcript has + caught up, and the new-workspace composer until the branch list has loaded and the draft is + saved. Enter pressed before that refuses the send. +- After an action that opens a view, wait for the view's text (`agent-browser wait --text ""`) + before the next key press. The URL changes before the view mounts, and an Escape sent too early + leaves a dialog open that hides the rest of the page from `find`. +- Chain a wait and the key presses after it with `&&`. A failed wait then stops the chain, and no + keys go to whatever has focus instead. +- `snapshot -i` lists only interactive elements. Use a full `snapshot` to read text, headings and + states such as `INTERRUPTED`. +- Mock replies are `Mock response: `. `[mock:long-stream]` in a message streams for + about 20 s, long enough to press Stop. Markers such as `[mock:error:api]` and + `[mock:list-languages]` are in `src/node/services/mock/mockAiRouter.ts`. Mock tool calls are + canned and leave the worktree alone. A canned `todo_write` (`[mock:todo-nudge]`) still writes + its TODO list to the workspace's session folder. +- The CLI drives the same server. Run it under `env -i`: the live Xum's server URL and auth + token are in your env under both `XUM_*` and legacy `MUX_*` names, and the CLI reads either. + With them gone it finds the sandbox through `XUM_ROOT`: + + ```bash + env -i PATH="$PATH" HOME="$HOME" XUM_ROOT="$VERIFY_XUM_ROOT" \ + node "$VERIFY_XUM_CHECKOUT/dist/cli/api.mjs" workspace list + ``` + +## Evidence + +Evidence goes to `$VERIFY_XUM_EVIDENCE`, which is `$VERIFY_XUM_HOME//evidence`. +`VERIFY_XUM_HOME` defaults to `$XUM_SCRATCH_DIR/verify-xum`, else `$XDG_RUNTIME_DIR/verify-xum`, +else `$HOME/.cache/verify-xum`, and never to a shared TMPDIR. Launch creates it with mode 0700. +Launch and every command that reads a run refuse a home inside the checkout, inside the main +worktree of a linked checkout, or inside the workspace root that the checkout's Vite reports +(`searchForWorkspaceRoot`, which can be a parent folder), symlinks and `..` resolved: Vite serves +every file there to anyone who reaches its port, the run's token copy and evidence included. +They also refuse when that Vite cannot report its root (run `bun install`). +Every command that reads a run's `state.env` first checks that the home and the run folder are +real folders (no symlinks), owned by you and not writable by group or others, and refuses +otherwise: the script sources that file. Cleanup never +deletes the evidence or the server log next to it (`/server.log`). Evidence is meant to be +shared: `capture` replaces the backend's auth token (and any `token=` URL value) with +`` in the URL and snapshot it writes, and cleanup does the same in every text file of +the evidence and in the server log. A screenshot is not redacted, so do not capture a page that +shows the token. + +```bash +.xum/skills/verify-xum/scripts/verify-xum.sh capture chat/reply +``` + +`capture /` writes `.png`, `.aria.txt` (full ARIA snapshot) and +`.url.txt` under `evidence//`. Pass absolute paths to any other `agent-browser` +output: it resolves relative paths against its daemon's directory. + +Proof standards: + +- Drive the path a user takes. No test-only routes, no state written behind the UI's back except + the seed, and edits to the worktree that stand in for the user's own editor. +- Mock only at a production boundary. The mock AI stands in for the model provider. Do not stub + Xum's own code, routes or stores to make a step pass. +- Name what the mock skips. It never calls a provider, and its tool calls are canned and leave + the worktree alone. A mock run does not prove a change to provider requests or tool execution: + report that part as not verified, or prove it at that boundary. +- Capture the action and the resulting state, not only the final screen. +- Check side effects next to what is visible: rows in + `$VERIFY_XUM_ROOT/sessions//chat.jsonl`, entries in `$VERIFY_XUM_ROOT/config.json`, + git state in the worktree, or the CLI view above. +- Report an entry point you could not drive with the command and the unmet precondition. Do not + count it as verified through another path. + +## Cleanup + +```bash +.xum/skills/verify-xum/scripts/verify-xum.sh cleanup +``` + +Cleanup closes this run's browser session, stops the process group that launch started (SIGTERM, +then SIGKILL after 20 s), deletes the sandbox XUM_ROOT, the run's `tmp` folder (the server tree's +TMPDIR, where the sandbox makes its XUM_ROOT) and the demo repo, and prints the evidence +path and file count. It refuses a run that another checkout launched. It signals the group only +while one of its processes carries the run's id and a live group leader is still the launch's +`make dev-server-sandbox`, so it kills nothing it did not start. Then it blanks the group id in +the run's `state.env`. Run it after every run, failed ones too. `doctor`, `env`, `open`, +`workspace-field`, `capture`, `idle`, `send-ready` and `cleanup` take an optional run directory +after their other arguments. Without one they use the newest launch from this checkout. + +## Gotchas + +- The bash tool's environment carries the live Xum's `PORT`, auth token and provider keys. Launch + drops them with `env -i`. Never start a server or CLI by hand without the same care. +- The first open of a lazily loaded panel on a cold Vite cache (Review, for one) reloads the page: + `server.log` shows `optimized dependencies changed. reloading`. The panel and its tab stay open + across the reload. Run `idle`, wait for the panel's text, and continue from the open tab (see + the Review recipe). +- Now and then a lazily loaded module request from agent-browser to Vite never completes. The + page or panel then shows ` failed to load.` with a `Reload` button, and the console logs + `Failed to fetch dynamically imported module`. curl gets the same module at once, so it is not + a product failure. `verify-xum.sh idle` exits 3 on it. Run `agent-browser reload`, then repeat + the step, and note the reload in your report. +- A page that stays on `Loading Xum` under host load recovers with `agent-browser reload`. +- agent-browser's headless Chromium matches `(hover: none)`, so Tailwind's `hover:` and + `group-hover:` variants never apply. Controls that appear on hover (a sidebar row's archive + icon) ignore clicks there. Focus them (`agent-browser focus '[aria-label=""]'`) and press + Enter, which is the keyboard user's path. +- `agent-browser` sessions are host-global. Use only the run's `AGENT_BROWSER_SESSION`. +- HOME is the real one, so git inside the app uses your global git config. Xum writes its own + state only under the sandbox XUM_ROOT. diff --git a/.xum/skills/verify-xum/features/README.md b/.xum/skills/verify-xum/features/README.md new file mode 100644 index 00000000000..f27dd13b5b6 --- /dev/null +++ b/.xum/skills/verify-xum/features/README.md @@ -0,0 +1,87 @@ +# Xum verification map + +One recipe per user-facing feature of the Xum web UI. Read this index, then drive the matching +feature file with the `verify-xum` harness. Update the file in the same change as the feature. + +## Baseline preconditions + +- `verify-xum.sh launch` started the run, and `verify-xum.sh doctor` prints only `ok` lines. +- The seed holds the trusted project `demo-app` (one commit on `main`: `README.md`, `src/math.ts`) + and the workspace `Verify playground` (branch `verify-playground`, id `$VERIFY_XUM_WORKSPACE_ID`, + worktree `$VERIFY_XUM_WORKTREE`) with an empty transcript. +- Onboarding is marked seen, tutorials are off, and a fake Anthropic provider is configured, so + the composer sends. The mock AI answers every turn. + +## Driving conventions + +- Start each bash call with `eval "$(.xum/skills/verify-xum/scripts/verify-xum.sh env)"`. It also + puts the helper on `PATH`, so the recipes call it as `verify-xum.sh`. +- Run each step's commands as one `&&` chain, or under `set -e`, so the first failed `wait` stops + the step. Keys pressed after a failed wait go to whatever has focus. In one run a palette query + went into the composer, and the Enter after it sent that text as a chat message. +- Start every recipe whose preconditions name `Verify playground` by opening it, even after + another recipe: the workspaces and command-palette recipes end on `release-notes`, and the + recipes read evidence from `$VERIFY_XUM_WORKSPACE_ID`. Open the app with + `verify-xum.sh open "/workspace/$VERIFY_XUM_WORKSPACE_ID"` and run `verify-xum.sh idle`. A new + browser session starts with the project collapsed, and the recipes choose workspace rows by + name, so expand it when it is collapsed. This form fails only when the button is there and the + click fails, so it is safe in an `&&` chain: + + ```bash + { [[ $(agent-browser eval '!!document.querySelector("[aria-label=\"Expand project demo-app\"]")') != true ]] \ + || agent-browser find role button click --name "Expand project demo-app" --exact; } + ``` + +- Open pages with `verify-xum.sh open `. It adds the backend's auth token without putting + it on any command line (SKILL.md Drive). +- Read a workspace's id, or another field of its `config.json` entry, with + `verify-xum.sh workspace-field `, for example + `RN_ID=$(verify-xum.sh workspace-field release-notes id)`. The app makes up ids at create time, + and a shell variable is gone in the next bash call, so read it in each call that needs it. +- Send a message in three steps: fill `Message`, run `verify-xum.sh send-ready`, then press Enter + or choose `Send message`. +- Use accessible names with `agent-browser find role --name "" --exact`. + Names are exact strings from the UI. `find text "" click --exact` is the fallback for rows + without a role. +- `find` does not wait. After a step that navigates or opens a view, the recipe waits for the next + target first, with `agent-browser wait '[aria-label=""]'` or + `agent-browser wait --text ""`. Keep those waits when you adapt a recipe. +- When a wait times out, run `verify-xum.sh idle`. Exit 3 means a module failed to load (SKILL.md + Gotchas): run `agent-browser reload`, repeat the step, and note the reload with the evidence. +- Run recipes in the order of this index when you drive more than one. A recipe that changes a + setting restores it at its end. +- Shortcuts are the Linux bindings that agent-browser presses. `src/browser/utils/ui/keybinds.ts` + defines them, with their macOS variants. + +## Proof and skip reporting + +- Capture with `verify-xum.sh capture /`. The feature part of the name is the file + name below. +- Capture the state after each action that the recipe marks as proof, not only the last screen. +- Check the side effect the recipe names (`chat.jsonl`, `config.json`, git state, or the CLI) + and save its output next to the captures, as `$VERIFY_XUM_EVIDENCE//.txt`. Run + `mkdir -p "$VERIFY_XUM_EVIDENCE/"` first: only `capture` creates that folder. +- Report an entry point you could not drive with the command you ran and the unmet precondition. + +## Feature entry contract + +Each feature file starts with an H1 title and one paragraph on the user-visible behavior. Then it +has exactly four H2 sections in this order: + +1. `Sub-features` lists short IDs with one line each. +2. `How to get to it (user POV)` lists every entry point a user has. +3. `Driving it with agent-browser` starts with `Preconditions:` and pairs each user action with + the exact command and the observable result. +4. `Gotchas` lists traps that waste or invalidate a run. + +## Features + +- [Workspaces](./workspaces.md) covers creating a workspace from the project composer, opening it + from the sidebar, archiving and restoring it, and the CLI view of the list. +- [Chat](./chat.md) covers sending a message, the streamed reply, Stop, and the transcript after a + reload. +- [Review](./review.md) covers the Review tab's diff of uncommitted worktree changes, marking a + hunk as read, and the sidebar's keyboard toggle. +- [Command palette](./command-palette.md) covers opening the palette, switching workspaces, and + running a command. +- [Settings](./settings.md) covers opening Settings, changing a preference, and its persistence. diff --git a/.xum/skills/verify-xum/features/chat.md b/.xum/skills/verify-xum/features/chat.md new file mode 100644 index 00000000000..64d5e6f2a3e --- /dev/null +++ b/.xum/skills/verify-xum/features/chat.md @@ -0,0 +1,68 @@ +# Chat + +The user types a message in a workspace's composer and sends it. The reply streams into the +`Conversation transcript` log, the user can stop a reply that is still streaming, and the +transcript is the same after a reload. + +## Sub-features + +- `chat-send` sends a message and shows the user row and the reply. +- `chat-stop` stops a streaming reply and marks it interrupted. +- `chat-persist` shows the same transcript after a page reload, from `chat.jsonl`. + +## How to get to it (user POV) + +- Type in the `Message` textbox of an open workspace and press Enter, or choose `Send message`. +- Choose `Stop streaming` while a reply streams. +- Reload the page on `/workspace/`. + +## Driving it with agent-browser + +Preconditions: + +- Baseline state. The app is open on `Verify playground` with an empty transcript. +- `verify-xum.sh idle` returned. + +- **Send with Enter.** Run + `agent-browser find role textbox fill "Hello from verify-xum" --name "Message" --exact`, + `verify-xum.sh send-ready`, `agent-browser press Enter`, + `agent-browser wait --text "Mock response: Hello from verify-xum"` and `verify-xum.sh idle`. The + transcript shows the user row `Hello from verify-xum` and the reply + `Mock response: Hello from verify-xum`. Capture `chat/reply`. +- **Send with the button.** Fill `Second message`, run `verify-xum.sh send-ready`, then run + `agent-browser find role button click --name "Send message" --exact` and + `agent-browser wait --text "Mock response: Second message"`. +- **Check history.** The transcript shows a reply before the server writes it to `chat.jsonl` at + the end of the stream. Wait for the saved reply, then save the last four rows of the workspace + history: + + ```bash + chat_jsonl="$VERIFY_XUM_ROOT/sessions/$VERIFY_XUM_WORKSPACE_ID/chat.jsonl" + timeout 10 sh -c 'until grep -qF "Mock response: Second message" "$1"; do sleep 0.2; done' sh "$chat_jsonl" + mkdir -p "$VERIFY_XUM_EVIDENCE/chat" + tail -n 4 "$chat_jsonl" >"$VERIFY_XUM_EVIDENCE/chat/history.txt" + ``` + + The file has two `"role":"user"` rows and two `"role":"assistant"` rows with the reply texts. + +- **Stop a stream.** Fill `[mock:long-stream] tell a long story`, run `verify-xum.sh send-ready` + and press Enter. Run `agent-browser wait '[aria-label="Stop streaming"]'`, then wait for a few + chunks with + `agent-browser wait --fn "document.body.innerText.split('Streaming response').length > 40"`. + Run `agent-browser click '[aria-label="Stop streaming"]'` and `verify-xum.sh idle`. The reply + keeps its partial text, an `INTERRUPTED` divider follows it, and the button + `Continue interrupted response` appears. Capture `chat/stopped`. +- **Reload.** Run `agent-browser reload` and `verify-xum.sh idle`. The transcript still shows both + replies, the partial reply and `INTERRUPTED`. Capture `chat/reloaded`. + +## Gotchas + +- Enter pressed before `send-ready` returns is refused with the alert + `Transcript is still loading — wait until it is up to date before sending or editing.` The draft + stays in the box, and the alert stays until you choose `Dismiss`. +- `verify-xum.sh idle` returns at once before a stream starts. Wait for the reply text or the + `Stop streaming` button first, then for idle. +- The mock writes partial text to disk every 100 ms. A Stop in the first 100 ms of a stream can + leave `INTERRUPTED` with no text after a reload. Wait for chunks before you stop. +- A model picked in the composer stays in memory until a send. A reload before the send shows the + workspace's saved model again (`src/browser/utils/modelChange.ts`). diff --git a/.xum/skills/verify-xum/features/command-palette.md b/.xum/skills/verify-xum/features/command-palette.md new file mode 100644 index 00000000000..80592a98c09 --- /dev/null +++ b/.xum/skills/verify-xum/features/command-palette.md @@ -0,0 +1,64 @@ +# Command palette + +The command palette is a search box over workspaces and commands. Without a prefix it lists +workspaces to switch to. With `>` it lists commands, such as `Change Model…` and `Open Settings`. + +## Sub-features + +- `palette-open` opens the palette with each shortcut and closes it with Escape. +- `palette-switch` switches to another workspace by typing part of its title. +- `palette-command` runs a command from `>` mode. + +## How to get to it (user POV) + +- Press Ctrl+Shift+P (`OPEN_COMMAND_PALETTE`) for workspace search. +- Press F4 (`OPEN_COMMAND_PALETTE_ACTIONS`) for command mode, which prefills `>`. + +## Driving it with agent-browser + +Preconditions: + +- Baseline state plus a second workspace `release-notes` (create it with the + [workspaces](./workspaces.md) recipe, or skip `palette-switch` and report it). +- The app is open on `Verify playground`, and `verify-xum.sh idle` returned. + +- **Open and close.** Run `agent-browser press Control+Shift+P`, + `agent-browser wait '[aria-label="Command palette"]'` and + `agent-browser wait '[role="listbox"][aria-label="Suggestions"] [role="option"]'`: on the first + open of a page the list fills a moment after the combobox shows. The combobox + `Command palette` is expanded, with the listbox `Suggestions`. Run `agent-browser press Escape`. + The combobox is gone. +- **Switch workspace.** Read the id with `RN_ID=$(verify-xum.sh workspace-field release-notes id)` + (see [workspaces](./workspaces.md) **Open from the sidebar**). Press Ctrl+Shift+P, wait for the + palette as above, then run + `agent-browser keyboard type "release"` and + `agent-browser wait '[role="option"][aria-selected="true"][aria-label="release-notes"]'`. + `Suggestions` lists the option `release-notes`, selected. Run `agent-browser press Enter` and + `agent-browser wait --url "**/workspace/$RN_ID"`: the palette closes first and navigates a + moment later. The URL is `/workspace/$RN_ID`. Capture + `command-palette/switched`. +- **Command mode.** Run `agent-browser press F4` and + `agent-browser wait '[aria-label="Command palette"]'`. The combobox value is `>`, and + `Suggestions` lists commands such as `Open Settings` and `Change Model…`. +- **Run a command.** Run `agent-browser keyboard type "Change Model"`, + `agent-browser wait '[role="option"][aria-selected="true"][aria-label="Change Model…"]'`, + `agent-browser press Enter` and `agent-browser wait --text "Show all models…"`. The model picker + opens with the textbox `Search [provider:model-name]`. Run + `agent-browser find text "Sonnet 5.5" click --exact`. The composer's combobox reads + `Model: Sonnet 5.5`. Capture `command-palette/model`. + +## Gotchas + +- Wait for the option you want to be selected before you press Enter. The first palette open after + a page load shows a plain input while the palette's code loads (`LoadingPaletteInput` at + `src/browser/components/CommandPalette/LazyCommandPalette.tsx:38`), with the same label + `Command palette`. A snapshot shows it as `textbox "Command palette"` until the code loads, then + as `combobox "Command palette"`. agent-browser types and presses Enter in a few milliseconds, before the code + arrives. Since #7897 that input keeps the Enter and runs it on the first option once the code + loads. Before it, the Enter was lost. The wait proves that the option Enter runs is the one the + recipe expects. +- Type into the palette with `agent-browser keyboard type`. The combobox has focus as soon as it + opens. +- Workspace options show the title only. A leading `•` marks the open workspace. +- The model pick stays in memory until a send (see [chat](./chat.md) Gotchas). Do not reload + between `Change Model…` and the check. diff --git a/.xum/skills/verify-xum/features/review.md b/.xum/skills/verify-xum/features/review.md new file mode 100644 index 00000000000..43543ab9f45 --- /dev/null +++ b/.xum/skills/verify-xum/features/review.md @@ -0,0 +1,101 @@ +# Review + +The Review tab in the right sidebar shows the workspace's diff against its base branch, one hunk +per change. The user includes uncommitted edits, reads the diff, and marks a hunk as read. The tab +label counts read hunks against all hunks. + +## Sub-features + +- `review-open` opens the Review tab from the right sidebar. +- `review-uncommitted` shows uncommitted worktree edits once `Uncommitted:` is checked. +- `review-read` marks a hunk as read and updates the `Review /` count. +- `review-toggle` hides and shows the right sidebar with the keyboard and keeps the open tab. + +## How to get to it (user POV) + +- Choose `Expand right sidebar`, then choose `Review` under `Open a tool`. +- Press Ctrl+Shift+B to collapse the right sidebar and to open it again on its last tab. + +## Driving it with agent-browser + +Preconditions: + +- Baseline state. The worktree `$VERIFY_XUM_WORKTREE` has no changes. +- The app is open on `Verify playground`, the right sidebar is collapsed with no tab open, and + `Uncommitted:` is unchecked (`userPreferences.review.includeUncommitted` is not `true` in + `config.json`). The recipe's Restore step leaves this state. + +- **Edit a file.** Change the worktree the way an editor would: + + ```bash + printf 'export function sub(a: number, b: number): number {\n return a - b;\n}\n' >>"$VERIFY_XUM_WORKTREE/src/math.ts" + ``` + + `git -C "$VERIFY_XUM_WORKTREE" status --short` prints ` M src/math.ts`. + +- **Open the sidebar.** Run + `agent-browser find role button click --name "Expand right sidebar" --exact`, + `agent-browser wait --text "Open a tool"` and `verify-xum.sh idle`. The heading `Open a tool` + lists a `Review` button. +- **Open Review.** Run `agent-browser find role button click --name "Review" --exact` and + `agent-browser wait --text "Uncommitted:"`. The tab `Review Close Review` is selected, and the + panel reads `No changes found` with `Uncommitted:` unchecked. If the page reloads here instead + (a cold Vite cache, see Gotchas), the sidebar and the Review tab stay open. Run + `verify-xum.sh idle` and `agent-browser wait --text "Uncommitted:"`, then continue at + **Include uncommitted edits**. +- **Include uncommitted edits.** Run + `agent-browser find role checkbox check --name "Uncommitted:" --exact` and + `agent-browser wait --text "math.ts"`. The file tree lists `src` and `math.ts` with `+3`, the + hunk shows the three added lines of `sub`, and the tab reads `Review 0/1`. Capture + `review/diff`. +- **Mark as read.** Run `verify-xum.sh idle` and + `agent-browser wait '[aria-label="Mark as read (M)"]'` (after a reload the hunk's button shows + late), then `agent-browser find role button click --name "Mark as read (M)" --exact`. The tab + reads `Review 1/1`, and the hunk reads `Hunk marked as read`. + The app writes `review-state.json` a moment later. Wait for it: + + ```bash + timeout 10 sh -c 'until grep -qsF "\"isRead\": true" "$1"; do sleep 0.2; done' sh "$VERIFY_XUM_ROOT/sessions/$VERIFY_XUM_WORKSPACE_ID/review-state.json" + ``` + + `$VERIFY_XUM_ROOT/sessions/$VERIFY_XUM_WORKSPACE_ID/review-state.json` has `"isRead": true` for + that hunk. Capture `review/read`. + +- **Keyboard toggle.** Run `agent-browser press Control+Shift+B`. The sidebar collapses, and the + button `Expand right sidebar` is back. Run `agent-browser press Control+Shift+B` again. The + button reads `Collapse right sidebar`, and the tab `Review 1/1 Close Review` is selected. +- **Restore.** The sidebar slid open again: run `verify-xum.sh idle`, then + `agent-browser find role button click --name "Mark as read (M)" --exact`. The tab reads + `Review 0/1` at once, and the app writes `review-state.json` a moment later. Wait for + it: + + ```bash + timeout 10 sh -c 'while grep -qF "\"isRead\": true" "$1"; do sleep 0.2; done' sh "$VERIFY_XUM_ROOT/sessions/$VERIFY_XUM_WORKSPACE_ID/review-state.json" + ``` + + `review-state.json` has no `"isRead": true`. Run + `agent-browser find role checkbox click --name "Uncommitted:" --exact`. The box is + `checked=false` at once. Wait until the server has written it: + + ```bash + timeout 10 sh -c 'until grep -qF "\"includeUncommitted\": false" "$1"; do sleep 0.2; done' sh "$VERIFY_XUM_ROOT/config.json" + ``` + + `config.json` has `"includeUncommitted": false` under `userPreferences.review`. Run `git -C "$VERIFY_XUM_WORKTREE" checkout -- src/math.ts`, + `agent-browser find role button click --name "Close Review" --exact` and + `agent-browser wait '[aria-label="Expand right sidebar"]'`. Closing the last tab collapses the + sidebar. + +## Gotchas + +- `find role ... --name` matches substrings without `--exact`. The hunk itself is a button whose + name starts with `Mark as read (M)`, so the call without `--exact` selects the hunk and marks + nothing. +- On a cold Vite cache the first open of Review reloads the page (see SKILL.md Gotchas). The + sidebar and the Review tab stay open after the reload, so the recipe goes on from the open tab. +- A click on `Review` while the sidebar still slides open is lost. Keep the `idle` call before it. +- Skip a Restore step and a rerun in the same run fails. A hunk's id comes from its content, so + the same edit comes back already read. With `Uncommitted:` already checked, the panel did not + show a new edit within 25 s, until `Refresh diff`. With the Review tab open, + `Expand right sidebar` opens that tab instead of `Open a tool`. +- `find` has no `uncheck` action. `find role checkbox click` toggles the box. diff --git a/.xum/skills/verify-xum/features/settings.md b/.xum/skills/verify-xum/features/settings.md new file mode 100644 index 00000000000..72b8b634138 --- /dev/null +++ b/.xum/skills/verify-xum/features/settings.md @@ -0,0 +1,84 @@ +# Settings + +Settings is a dialog over the app with one section per area (General, Providers, Models, and +more). Each section has its own route under `/settings/`. A change applies at once and persists in +the server's `config.json`. + +## Sub-features + +- `settings-open` opens the dialog from the sidebar button, its shortcut, the palette, and its route. +- `settings-toggle` changes the `Flat chat list` switch and shows its effect on the sidebar. +- `settings-persist` keeps the change across a reload and in `config.json`. + +## How to get to it (user POV) + +- Choose `Open settings` (the gear in the sidebar header). +- Press Ctrl+, (`OPEN_SETTINGS`). +- Press F4, type `Open Settings`, and press Enter. +- Open `$VERIFY_XUM_URL/settings/general`. + +## Driving it with agent-browser + +Preconditions: + +- Baseline state. `Flat chat list` is off. +- The app is open on `Verify playground`. + +- **Open from the button.** Run + `agent-browser find role button click --name "Open settings" --exact` and + `agent-browser wait --text "Choose your preferred theme"`. The URL is `/settings/general`, and a + full `snapshot` shows the dialog `Settings` with the section buttons `General`, `Providers` and + `Models`. Run `agent-browser press Escape` and + `agent-browser wait --fn '!document.querySelector("[data-modal-dialog-overlay][data-state=open], [role=dialog][aria-modal=true]")'`. + The URL is back on the workspace. +- **Open with the keyboard.** Run `agent-browser press Control+,` and + `agent-browser wait --text "Choose your preferred theme"`. The URL is `/settings/general`. Close + it with Escape and the same wait as above. +- **Open from the palette.** Run `agent-browser press F4`, + `agent-browser wait '[aria-label="Command palette"]'`, + `agent-browser keyboard type "Open Settings"`, + `agent-browser wait '[role="option"][aria-selected="true"][aria-label="Open Settings"]'`, + `agent-browser press Enter` and `agent-browser wait --text "Choose your preferred theme"`. The + URL is `/settings/general`. +- **Change a setting.** Run + `agent-browser find role switch click --name "Toggle flat chat list" --exact`. The switch is + `checked=true` at once, but the server writes `config.json` a moment later. Wait for the write + before you capture or reload: + + ```bash + timeout 10 sh -c 'until grep -qF "\"sidebarFlatMode\": true" "$1"; do sleep 0.2; done' sh "$VERIFY_XUM_ROOT/config.json" + ``` + + `$VERIFY_XUM_ROOT/config.json` has `"sidebarFlatMode": true` under `userPreferences.ui`. Capture + `settings/flat-on`. + +- **See the effect.** Run `agent-browser press Escape` and `agent-browser wait --text "RECENT"`. + The sidebar has a `RECENT` group, and the workspace row is + `Select workspace Verify playground (demo-app)`. +- **Reload.** Run `agent-browser reload`, + `verify-xum.sh open /settings/general` and + `agent-browser wait --text "Choose your preferred theme"`. The switch `Toggle flat chat list` is + still `checked=true`. Capture `settings/persisted`. +- **Restore.** Run + `agent-browser find role switch click --name "Toggle flat chat list" --exact`, wait until + `config.json` has `"sidebarFlatMode": false` (the same `timeout 10` loop with `false`), and run + `agent-browser press Escape`. The page opened + Settings by its URL, so Escape lands on `/project?...`. Run + `verify-xum.sh open "/workspace/$VERIFY_XUM_WORKSPACE_ID"` and + `verify-xum.sh idle` to be back on `Verify playground`. + +## Gotchas + +- The palette shortcuts do nothing while a modal dialog is open, and the Settings dialog stays in + the page for a moment after Escape (`isDialogOpen()` at `src/browser/utils/ui/keybinds.ts:291`, + checked by the shortcut handler at `src/browser/App.tsx:1133`). The wait after Escape uses the same + selectors. Without it, F4 is lost and the typed text goes to the composer. +- The palette steps wait for the selected option before Enter. See + [command-palette](./command-palette.md) Gotchas. +- `snapshot -i` lists only the dialog's buttons and switches. Use a full `snapshot` to read the + section text. +- Flat mode renames sidebar rows to `Select workspace (<project>)`. Restore it before you + drive another recipe. +- Read where a control saves before you name its side effect. `Flat chat list` calls + `updateUserPreferences` (`src/browser/features/Settings/Sections/GeneralSection.tsx`), and other + controls there call other routes. diff --git a/.xum/skills/verify-xum/features/workspaces.md b/.xum/skills/verify-xum/features/workspaces.md new file mode 100644 index 00000000000..26356f57ccd --- /dev/null +++ b/.xum/skills/verify-xum/features/workspaces.md @@ -0,0 +1,127 @@ +# Workspaces + +A workspace is one chat with its own git worktree and branch of a project. The user creates one +from the project composer by naming it and sending a first message, opens it from the sidebar, +archives it to hide it, and restores it from the project's archived list. + +## Sub-features + +- `ws-create` creates a named workspace, its worktree and branch, and sends the first message. +- `ws-open` opens a workspace from its sidebar row. +- `ws-archive` hides a workspace from the sidebar and records the archive time. +- `ws-restore` brings an archived workspace back from the project page. +- `ws-cli` lists the same workspaces through `xum api workspace list`. + +## How to get to it (user POV) + +- Choose `New chat in demo-app` on the project's sidebar row, or press Ctrl+N. +- Choose a `Select workspace <title>` row in the sidebar. +- Hover a sidebar row and choose its `Archive workspace <title>` icon, or tab to the icon and + press Enter, or press Ctrl+Shift+Backspace in the open workspace. +- Choose the project row (`Create workspace in demo-app`), then `Expand archived workspaces` and + `Restore workspace <title>`. +- Run `xum api workspace list` against the server. + +## Driving it with agent-browser + +Preconditions: + +- Baseline state. No workspace is named `release-notes` or `scratch-notes`. +- The app is open on `Verify playground` with the project expanded (see the + [driving conventions](./README.md#driving-conventions)), and `verify-xum.sh idle` returned. + +- **Open the composer.** Choose `New chat in demo-app`. Run + `agent-browser find role button click --name "New chat in demo-app" --exact` and + `agent-browser wait '[aria-label="Workspace name"]'`. The URL is + `$VERIFY_XUM_URL/project?project=demo-app-<hash>&draft=<id>`, and the page has the textboxes + `Workspace name` and `Message`. +- **Create.** Name the workspace and send. Run + `agent-browser find role textbox fill "release-notes" --name "Workspace name" --exact`, + `agent-browser find role textbox fill "Draft the release notes" --name "Message" --exact`, + `verify-xum.sh send-ready`, + `agent-browser find role button click --name "Send message" --exact` and + `agent-browser wait --text "Mock response: Draft the release notes"` and `verify-xum.sh idle`. + The URL changes to `/workspace/<new id>`, and the sidebar has a `Select workspace release-notes` + row. Capture `workspaces/created`. +- **Check the worktree.** Run `git -C "$VERIFY_XUM_PROJECT" worktree list`. It lists + `$VERIFY_XUM_ROOT/src/demo-app/release-notes` on branch `release-notes`. +- **Keyboard create.** Press Ctrl+N. Run `agent-browser press Control+n` and + `agent-browser wait '[aria-label="Workspace name"]'`. The URL changes to + `/project?project=demo-app-<hash>&draft=<id>`. Create `scratch-notes` the same way as above, + with the message `Collect scratch notes`, and run `verify-xum.sh idle` after its reply. +- **Open from the sidebar.** The app made up the workspace id at create time. Read it from + `config.json` in each bash call that needs it: + `RN_ID=$(verify-xum.sh workspace-field release-notes id)`. Run + `agent-browser find role button click --name "Select workspace release-notes" --exact` and + `agent-browser wait --url "**/workspace/$RN_ID"`. The URL is `/workspace/$RN_ID`. Run + `agent-browser find role button click --name "Select workspace Verify playground" --exact` and + `agent-browser wait --url "**/workspace/$VERIFY_XUM_WORKSPACE_ID"`. The URL is + `/workspace/$VERIFY_XUM_WORKSPACE_ID`. +- **Archive from the row.** Focus the row's archive icon and press Enter. Run + `agent-browser wait '[aria-label="Archive workspace release-notes"]'`, + `agent-browser focus '[aria-label="Archive workspace release-notes"]'` and + `agent-browser press Enter`. The archive request is still running when Enter returns. Wait for + its result with + `agent-browser wait --fn '!document.querySelector("[aria-label=\"Select workspace release-notes\"]")'`. + The `Select workspace release-notes` row is gone. Wait until the server has written the + archive: + + ```bash + timeout 10 sh -c 'until [ -n "$(verify-xum.sh workspace-field release-notes archivedAt)" ]; do sleep 0.2; done' + ``` + + In `$VERIFY_XUM_ROOT/config.json` the `release-notes` entry has an `archivedAt` time. Capture + `workspaces/archived`. + +- **Archive with the keyboard.** Open `scratch-notes` from its row and wait until the app is on + it: the shortcut archives whichever workspace is open. Run + `agent-browser find role button click --name "Select workspace scratch-notes" --exact`, + `agent-browser wait --url "**/workspace/$(verify-xum.sh workspace-field scratch-notes id)"` and + `verify-xum.sh idle`. Then press Ctrl+Shift+Backspace. Run + `agent-browser press Control+Shift+Backspace` and wait for the archive with + `agent-browser wait --fn '!document.querySelector("[aria-label=\"Select workspace scratch-notes\"]")'`. + The `Select workspace scratch-notes` row is gone. Wait for its `archivedAt` time with the same + `timeout 10` loop for `scratch-notes`. +- **Restore.** Open the project page and the archived list. Run + `agent-browser find role button click --name "Create workspace in demo-app" --exact`, + `agent-browser wait '[aria-label="Expand archived workspaces"]'`, + `agent-browser find role button click --name "Expand archived workspaces" --exact` and + `agent-browser wait '[aria-label="Restore workspace release-notes"]'`. The region + `Archived workspaces` lists `release-notes` and `scratch-notes`. Run + `agent-browser find role button click --name "Restore workspace release-notes" --exact`, then + wait for the result with `agent-browser wait --url "**/workspace/$RN_ID"` (`RN_ID` as in + **Open from the sidebar**) and + `agent-browser wait '[aria-label="Select workspace release-notes"]'`. The app is on + `/workspace/$RN_ID`, the sidebar row is back, and `config.json` has an + `unarchivedAt` time for it. Capture `workspaces/restored`. +- **CLI view.** List the workspaces through the CLI: + + ```bash + mkdir -p "$VERIFY_XUM_EVIDENCE/workspaces" + env -i PATH="$PATH" HOME="$HOME" XUM_ROOT="$VERIFY_XUM_ROOT" \ + node "$VERIFY_XUM_CHECKOUT/dist/cli/api.mjs" workspace list >"$VERIFY_XUM_EVIDENCE/workspaces/cli-list.txt" + ``` + + The file has the titles `Verify playground` and `release-notes`. + +## Gotchas + +- The reply text shows before its stream ends. Run `verify-xum.sh idle` after it. Archiving a + workspace that still streams opens the dialog `Archive "<title>" while streaming?`, and the + row stays. +- The project row's accessible name is `Create workspace in demo-app`, but choosing it opens the + project page. It creates nothing. +- Right after a create, the next new-workspace composer keeps `Send message` disabled for a few + seconds until its branch list has loaded and its draft is saved. `verify-xum.sh send-ready` + waits for it. +- With `Workspace name` empty, Xum names the workspace with random words (`curvy-mangos-pull`). + Fill the name so rows and branches have known names. +- Xum marks a reply read only while `document.hasFocus()` is true (`src/browser/App.tsx:1386`). + An earlier run saw it false in agent-browser's Chromium: a reply that finished while the + workspace was open left an unread status dot on its row, and the dot took the archive icon's + place. On agent-browser 0.33.2 (2026-11-03) `agent-browser eval 'document.hasFocus()'` printed + `true`, and a workspace watched until its reply kept its archive icon. If it prints `false`, + open the workspace once more before you archive it from its row, as the recipe does. +- A click on the archive icon selects the row instead in agent-browser's Chromium. The icon takes + clicks only on row hover, and that browser matches `(hover: none)`, so Tailwind's `group-hover` + never applies (see SKILL.md Gotchas). Focus the icon, as above. diff --git a/.xum/skills/verify-xum/scripts/verify-xum.sh b/.xum/skills/verify-xum/scripts/verify-xum.sh new file mode 100755 index 00000000000..dc01c324e75 --- /dev/null +++ b/.xum/skills/verify-xum/scripts/verify-xum.sh @@ -0,0 +1,732 @@ +#!/usr/bin/env bash +# One disposable, mock-AI Xum dev server per checkout, for verification runs (see ../SKILL.md). +# +# verify-xum.sh launch start and seed a sandbox, print its state +# verify-xum.sh doctor [RUN] read-only health check of a run +# verify-xum.sh env [RUN] print the run's `export` lines, put this script on PATH, clear git's +# repo variables (eval them) +# verify-xum.sh open PATH [RUN] open PATH of the run's UI in its browser session, token in no argv +# verify-xum.sh workspace-field NAME FIELD [RUN] print FIELD of workspace NAME from config.json +# verify-xum.sh capture NAME [RUN] screenshot + ARIA snapshot + URL into the run's evidence +# verify-xum.sh idle [RUN] wait until the page settles, exit 3 if a chunk failed to load +# verify-xum.sh send-ready [RUN] wait until the composer's Send button is enabled +# verify-xum.sh cleanup [RUN] stop the run's processes, delete its XUM_ROOT, keep evidence +# +# RUN defaults to the newest launch from this checkout. +set -euo pipefail + +# Git variables that point git at another repository (a hook's GIT_INDEX_FILE, an exported +# GIT_DIR, config from GIT_CONFIG_*) would make the checkout lookups and the demo repo's +# `git -C` commands below act on that repository: `git add .` would replace its index. This +# script clears them, and the `env` output clears them for the recipes' git commands too. The +# list is `git rev-parse --local-env-vars` plus the discovery and config variables. +# shellcheck disable=SC2046,SC2086 # lists of variable names +{ + GIT_REPO_ENV="GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_OBJECT_DIRECTORY + GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_COMMON_DIR GIT_IMPLICIT_WORK_TREE GIT_GRAFT_FILE + GIT_NO_REPLACE_OBJECTS GIT_REPLACE_REF_BASE GIT_SHALLOW_FILE GIT_PREFIX GIT_NAMESPACE + GIT_CEILING_DIRECTORIES GIT_DISCOVERY_ACROSS_FILESYSTEM ${!GIT_CONFIG*}" + unset $GIT_REPO_ENV + # Names a newer git adds to its own list. + GIT_REPO_ENV=$(printf '%s\n' $GIT_REPO_ENV $(git rev-parse --local-env-vars) | sort -u | tr '\n' ' ') + unset $GIT_REPO_ENV +} + +# With CDPATH set, `cd` to a relative folder can pick another folder and print its name, which +# the command substitutions below would capture. +unset CDPATH +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +CHECKOUT=$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel) +# Never a shared TMPDIR: another local account could create the folder first and swap a run's +# state.env, which this script sources (see require_private_dir). +VERIFY_XUM_HOME=${VERIFY_XUM_HOME:-${XUM_SCRATCH_DIR:+$XUM_SCRATCH_DIR/verify-xum}} +VERIFY_XUM_HOME=${VERIFY_XUM_HOME:-${XDG_RUNTIME_DIR:+$XDG_RUNTIME_DIR/verify-xum}} +VERIFY_XUM_HOME=${VERIFY_XUM_HOME:-$HOME/.cache/verify-xum} +# The run link in the git dir must point at an absolute path. +[[ $VERIFY_XUM_HOME == /* ]] || VERIFY_XUM_HOME="$PWD/$VERIFY_XUM_HOME" +CHECKOUT_GIT_DIR=$(git -C "$CHECKOUT" rev-parse --path-format=absolute --git-dir) +LAUNCH_GUARD="$CHECKOUT_GIT_DIR/verify-xum.lock.guard" +CURRENT_RUN_LINK="$CHECKOUT_GIT_DIR/verify-xum.current" +READY_TIMEOUT=${VERIFY_XUM_READY_TIMEOUT:-300} +DISCARD_BASE_URL=http://127.0.0.1:9/v1 + +die() { + echo "verify-xum: $*" >&2 + exit 1 +} + +need() { + local tool + for tool in "$@"; do + command -v "$tool" >/dev/null || die "missing tool: $tool" + done +} + +group_alive() { + [[ -n $1 ]] && kill -0 -- "-$1" 2>/dev/null +} + +# A process group belongs to this run only while one of its processes has +# VERIFY_XUM_SERVER_OF=<run id> in its env, so a group id reused after the run died is left alone. +# state.env never exports that name, so processes an agent starts after `eval "$(verify-xum.sh env)"` +# do not carry it. A live leader must also still be the `make dev-server-sandbox` that launch +# started. Linux reuses no pid while it names a process group, so a group whose leader exited +# keeps its id and stays ours. A zombie leader has an empty cmdline and counts as exited. +group_is_ours() { + local pid leader_cmdline + [[ -n $1 ]] || return 1 + if [[ -e /proc/$1 && $(ps -o stat= -p "$1" 2>/dev/null) != Z* ]]; then + leader_cmdline=$(tr '\0' ' ' 2>/dev/null <"/proc/$1/cmdline" || true) + [[ $leader_cmdline == *dev-server-sandbox* ]] || return 1 + fi + for pid in $(ps -o pid= -g "$1" 2>/dev/null); do + grep -qszx "VERIFY_XUM_SERVER_OF=$2" "/proc/$pid/environ" && return 0 + done + return 1 +} + +# A run's state.env is sourced, so the run folder and the home above it must be real folders (no +# symlinks), owned by this user and writable by nobody else. +require_private_dir() { + local dir=$1 mode + [[ -d $dir && ! -L $dir ]] || die "$dir is not a real folder" + [[ $(stat -c %u "$dir") == "$(id -u)" ]] || die "$dir is not owned by user $(id -u)" + mode=$(stat -c %a "$dir") + (((8#$mode & 8#022) == 0)) || die "$dir is writable by group or others (mode $mode)" +} + +# Every folder above a run home must be owned by root or this user and writable by nobody else, +# unless it is sticky like /tmp (as ssh's StrictModes checks): another account could otherwise +# rename the checked home away and put its own state.env in its place. Takes a physical path. +require_safe_ancestors() { + local dir=$1 owner mode + while [[ $dir != / ]]; do + dir=$(dirname "$dir") + owner=$(stat -c %u "$dir") + mode=$(stat -c %a "$dir") + [[ $owner == 0 || $owner == "$(id -u)" ]] || die "$dir, above the run home, is owned by user $owner" + (((8#$mode & 8#022) == 0 || (8#$mode & 8#1000) != 0)) \ + || die "$dir, above the run home, is writable by group or others and not sticky (mode $mode)" + done +} + +# A path without its trailing slashes: `-L dir/` follows the link, so `link//` would pass as a +# real folder. +without_trailing_slashes() { + local path=$1 + while [[ $path == ?*/ ]]; do path=${path%/}; done + echo "$path" +} + +# The folder Vite allows by default (server.fs.allow): the workspace root it finds at or above the +# checkout, which can be a parent folder (one whose package.json declares workspaces, for example). +# The checkout's own vite computes it from the physical checkout, as `vite` started there does, so +# this check cannot drift from Vite's rules. Fails closed: without an answer, no home is safe. +vite_workspace_root() { + local root + if root=$(cd -P -- "$CHECKOUT" && node --input-type=module -e ' + import { createRequire } from "node:module"; + import { pathToFileURL } from "node:url"; + const vite = createRequire(process.cwd() + "/package.json").resolve("vite"); + console.log((await import(pathToFileURL(vite).href)).searchForWorkspaceRoot(process.cwd())); + ') && [[ $root == /* && $root != *$'\n'* ]]; then + echo "$root" + else + die "the checkout's vite did not report its workspace root, so the run home cannot be checked against the folders Vite serves. Run \`bun install\` in $CHECKOUT." + fi +} + +# Vite serves every file under the checkout to whoever reaches its port (vite.config.ts denies only +# a few names), so a run home there would publish the run's token copy, its evidence and its +# sandbox. Refuses a home whose physical path (symlinks and .. resolved, even for parts that do +# not exist yet) is inside the checkout, inside the main worktree of a linked one, or inside +# Vite's workspace root. +require_home_outside_checkout() { # require_home_outside_checkout <home> + local home top common vite_root + home=$(realpath -m -- "$1") + common=$(git -C "$CHECKOUT" rev-parse --path-format=absolute --git-common-dir) + vite_root=$(vite_workspace_root) || exit 1 + for top in "$CHECKOUT" "$([[ $(basename "$common") == .git ]] && dirname "$common")" "$vite_root"; do + [[ -n $top ]] || continue + top=$(realpath -m -- "$top") + if [[ $top == / || $home == "$top" || $home == "$top"/* ]]; then + die "the run home $home is inside $top (the checkout, its main worktree or Vite's workspace root), where Vite serves every file to anyone who reaches its port (the run's token too). Set VERIFY_XUM_HOME to a folder outside it." + fi + done +} + +# The physical path of a run home, after checking it and every folder above it. Later reads use +# that path, so no symlink above the home is followed again. +checked_home() { + local home + home=$(without_trailing_slashes "$1") + [[ -d $home && ! -L $home ]] || die "$home is not a real folder" + home=$(cd -P -- "$home" && pwd -P) + require_home_outside_checkout "$home" + require_private_dir "$home" + require_safe_ancestors "$home" + echo "$home" +} + +# The run folder of a run argument, or behind the newest-run link (the one symlink allowed), +# checked before its state.env is read. +private_run_dir() { + local run home + run=$(without_trailing_slashes "$1") + [[ $run == "$CURRENT_RUN_LINK" ]] && run=$(readlink "$run") + home=$(checked_home "$(dirname "$run")") || exit 1 + run="$home/$(basename "$run")" + require_private_dir "$run" + echo "$run" +} + +# Opens the launch guard on fd 9 for flock. It sits in the checkout's git dir, which another +# account can write in a shared repository: a symlink planted there made `9>` truncate its target. +# The guard is created with O_EXCL (noclobber), which never follows a symlink, and opened read +# only, which can neither create nor truncate anything (flock needs no write access). Then fd 9 +# must be the regular file of this user that the path names. +open_launch_guard() { + [[ -e $LAUNCH_GUARD || -L $LAUNCH_GUARD ]] || (set -C && : >"$LAUNCH_GUARD") 2>/dev/null || true + [[ -f $LAUNCH_GUARD && ! -L $LAUNCH_GUARD ]] || die "$LAUNCH_GUARD is not a regular file (a symlink?). Remove it." + exec 9<"$LAUNCH_GUARD" + [[ $(stat -L -c '%d:%i:%u' /proc/self/fd/9) == "$(stat -c '%d:%i' -- "$LAUNCH_GUARD"):$(id -u)" ]] \ + || die "$LAUNCH_GUARD is not owned by user $(id -u), or it changed while it was opened. Remove it." +} + +# One server per checkout: two would share dist/ and node_modules/.vite. The guard is the newest +# run's process group, which records its own id in that run's state.env before it starts the +# server. The flock keeps two launches from passing the check at once, and it also covers the +# moment between the spawn and that record: the new group holds the flock until it has written +# its id, wherever the launcher dies. The server does not inherit it. +refuse_while_checkout_server_runs() { + open_launch_guard + # Part of the launch deadline: a short VERIFY_XUM_READY_TIMEOUT also bounds this wait. + flock -w "$(launch_seconds_left)" 9 || die "another verify-xum launch from this checkout holds $LAUNCH_GUARD" + [[ -f $CURRENT_RUN_LINK/state.env ]] || return 0 + local run + run=$(private_run_dir "$CURRENT_RUN_LINK") || exit 1 + ( + # shellcheck source=/dev/null + source "$run/state.env" + ! group_is_ours "$VERIFY_XUM_PGID" "$VERIFY_XUM_RUN_ID" + ) || die "the verify-xum run $(readlink "$CURRENT_RUN_LINK") still runs from this checkout. Run verify-xum.sh cleanup first." +} + +load_run() { + local run=${1:-$CURRENT_RUN_LINK} + [[ -f $run/state.env ]] || die "no run at $run (run: verify-xum.sh launch)" + run=$(private_run_dir "$run") || exit 1 + # shellcheck source=/dev/null + source "$run/state.env" + [[ $VERIFY_XUM_CHECKOUT == "$CHECKOUT" ]] \ + || die "run $run belongs to the checkout $VERIFY_XUM_CHECKOUT. Use that checkout's verify-xum.sh." +} + +LAUNCH_DEADLINE= +before_launch_deadline() { + ((SECONDS < LAUNCH_DEADLINE)) || die "launch did not finish in $READY_TIMEOUT s" +} + +# Seconds left before the launch deadline, at least 1: curl reads `--max-time 0` as no limit. +launch_seconds_left() { + local left=$((LAUNCH_DEADLINE - SECONDS)) + ((left > 0)) || left=1 + echo "$left" +} + +# A readiness probe that gives up at the launch deadline. +probe_before_launch_deadline() { + before_launch_deadline + curl -fs --max-time "$(launch_seconds_left)" -o /dev/null "$1" +} + +# A backend started with auth (#7904's `make dev-server`) records its token in server.lock. The +# header goes to curl on stdin, so the token stays out of every process's argv. +backend_curl() { + local token + token=$(json_field_or_empty "$VERIFY_XUM_ROOT/server.lock" token) + if [[ -n $token ]]; then + printf 'Authorization: Bearer %s\n' "$token" | curl -H @- "$@" + else + curl "$@" + fi +} + +api() { + before_launch_deadline + backend_curl -fsS --max-time "$(launch_seconds_left)" -X POST "$VERIFY_XUM_BACKEND/api/$1" \ + -H 'content-type: application/json' -d "$2" +} + +json_string() { + node -p 'JSON.stringify(process.argv[1])' "$1" +} + +json_field_or_empty() { + node -e ' + const fs = require("fs"); + try { + let v = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); + for (const k of process.argv[2].split(".")) v = v?.[k]; + if (v !== undefined && v !== null) console.log(String(v)); + } catch {}' "$1" "$2" +} + +backend_ready() { + node -e ' + const fs = require("fs"); + const [serverLock, buildStamp, launchMs] = process.argv.slice(1); + try { + const buildStampMs = Number(fs.readFileSync(buildStamp, "utf8")); + const serverStartedMs = Date.parse(JSON.parse(fs.readFileSync(serverLock, "utf8")).startedAt); + const builtSinceLaunch = buildStampMs >= Number(launchMs); + const serverRestartedAfterBuild = serverStartedMs > buildStampMs; + process.exit(builtSinceLaunch && serverRestartedAfterBuild ? 0 : 1); + } catch { + process.exit(1); + }' "$VERIFY_XUM_ROOT/server.lock" "$CHECKOUT/dist/.main-build-complete" "$1" \ + && probe_before_launch_deadline "$VERIFY_XUM_BACKEND/health" +} + +write_state() { + local name + for name in VERIFY_XUM_RUN_ID VERIFY_XUM_RUN VERIFY_XUM_CHECKOUT VERIFY_XUM_COMMIT VERIFY_XUM_PGID VERIFY_XUM_ROOT \ + VERIFY_XUM_BACKEND VERIFY_XUM_URL VERIFY_XUM_PROJECT VERIFY_XUM_WORKSPACE_ID \ + VERIFY_XUM_WORKTREE VERIFY_XUM_EVIDENCE AGENT_BROWSER_SESSION; do + printf 'export %s=%q\n' "$name" "${!name-}" + done >"$VERIFY_XUM_RUN/state.env.tmp" + mv -f "$VERIFY_XUM_RUN/state.env.tmp" "$VERIFY_XUM_RUN/state.env" +} + +check_bun() { + local want have + want=$(node -p 'require(process.argv[1]).packageManager.split("@")[1]' "$CHECKOUT/package.json") + have=$(bun --version) + [[ $have == "$want" ]] \ + || die "bun $have is first on PATH but package.json pins $want. Put Bun $want first on PATH (mise: PATH=\"\$(mise where bun@$want)/bin:\$PATH\")." +} + +create_demo_repo() { + local repo=$1 + mkdir -p "$repo/src" + printf '# Demo app\n\nA throwaway project for verify-xum.\n' >"$repo/README.md" + printf 'export function add(a: number, b: number): number {\n return a + b;\n}\n' >"$repo/src/math.ts" + git -C "$repo" init -q -b main + git -C "$repo" add . + git -C "$repo" -c user.name=verify-xum -c user.email=verify-xum@example.test \ + -c commit.gpgsign=false -c core.hooksPath=/dev/null commit -qm init +} + +seed_unreachable_fake_provider() { + api providers/setProviderConfig '{"provider":"anthropic","keyPath":["apiKey"],"value":"sk-ant-verify-xum-fake"}' >/dev/null + api providers/setProviderConfig "{\"provider\":\"anthropic\",\"keyPath\":[\"baseUrl\"],\"value\":\"$DISCARD_BASE_URL\"}" >/dev/null +} + +seed() { + local created status_file status + api splashScreens/markSplashScreenViewed '{"splashId":"onboarding-wizard-v1"}' >/dev/null + seed_unreachable_fake_provider + create_demo_repo "$VERIFY_XUM_PROJECT" + api projects/create "{\"projectPath\":$(json_string "$VERIFY_XUM_PROJECT")}" >/dev/null + api projects/setTrust "{\"projectPath\":$(json_string "$VERIFY_XUM_PROJECT"),\"trusted\":true}" >/dev/null + created="$VERIFY_XUM_RUN/scratch/workspace-create.json" + api workspace/create "{\"projectPath\":$(json_string "$VERIFY_XUM_PROJECT"),\"branchName\":\"verify-playground\",\"trunkBranch\":\"main\",\"title\":\"Verify playground\"}" >"$created" + VERIFY_XUM_WORKSPACE_ID=$(json_field_or_empty "$created" metadata.id) + [[ -n $VERIFY_XUM_WORKSPACE_ID ]] || die "workspace/create returned no id: $(head -c 400 "$created")" + status_file="$VERIFY_XUM_ROOT/sessions/$VERIFY_XUM_WORKSPACE_ID/init-status.json" + while status=$(json_field_or_empty "$status_file" status); [[ $status != success ]]; do + [[ $status == error ]] && die "workspace setup failed, see $status_file" + before_launch_deadline + sleep 0.5 + done + VERIFY_XUM_WORKTREE="$VERIFY_XUM_ROOT/src/$(basename "$VERIFY_XUM_PROJECT")/verify-playground" + [[ -d $VERIFY_XUM_WORKTREE ]] || die "no worktree at $VERIFY_XUM_WORKTREE" +} + +# nodemon waits for a file change after the backend crashes, and none comes during a launch. A +# crash before the first rebuild is expected (a stale dist/): the rebuild restarts it. +backend_crashed_after_rebuild() { + sed 's/\x1b\[[0-9;]*m//g' "$1" | awk ' + /\[nodemon\] starting `.*dist\/cli\/index\.js server/ && prefix == "" { prefix = $1 } + prefix != "" && $1 == prefix && /\[nodemon\] restarting due to changes/ { rebuilt = 1; crashed = 0 } + prefix != "" && $1 == prefix && /\[nodemon\] app crashed/ { crashed = rebuilt } + END { exit !crashed }' +} + +launch_failed() { + echo "verify-xum: launch failed. Server log tail ($VERIFY_XUM_RUN/server.log):" >&2 + tail -n 30 "$VERIFY_XUM_RUN/server.log" >&2 || true + if [[ -z $VERIFY_XUM_PGID ]]; then + # A signal can stop the launcher after the spawn and before the new group records its id. + # That group holds the launch flock until it has written the id or died, so take the flock + # anew and let cleanup read the id. Without it cleanup would drop the newest-run link while + # the server starts, and the next launch would start a second one. + exec 9>&- + open_launch_guard + if ! flock -w 10 9; then + echo "verify-xum: the new process group did not record its id in 10 s. The newest-run link stays: run verify-xum.sh cleanup." >&2 + return 0 + fi + fi + cmd_cleanup "$VERIFY_XUM_RUN" >&2 || true +} + +cmd_launch() { + local t0 backend_port vite_port log token + need git curl node bun make setsid flock ss realpath agent-browser + check_bun + LAUNCH_DEADLINE=$((SECONDS + READY_TIMEOUT)) + refuse_while_checkout_server_runs + + # Before any folder is made: a refused home creates nothing. + require_home_outside_checkout "$VERIFY_XUM_HOME" + if [[ ! -e $VERIFY_XUM_HOME ]]; then + # Parents this creates are private too, whatever the caller's umask. + (umask 077 && mkdir -p "$(dirname "$VERIFY_XUM_HOME")") + # Check the folders above before the home is made in them (the home itself does not exist yet). + require_safe_ancestors "$(cd -P -- "$(dirname "$VERIFY_XUM_HOME")" && pwd -P)/home" + mkdir -m 700 "$VERIFY_XUM_HOME" + fi + VERIFY_XUM_HOME=$(checked_home "$VERIFY_XUM_HOME") || exit 1 + VERIFY_XUM_RUN_ID="$(date -u +%Y%m%d-%H%M%S)-$$" + VERIFY_XUM_RUN="$VERIFY_XUM_HOME/$VERIFY_XUM_RUN_ID" + VERIFY_XUM_EVIDENCE="$VERIFY_XUM_RUN/evidence" + VERIFY_XUM_PROJECT="$VERIFY_XUM_RUN/scratch/demo-app" + VERIFY_XUM_CHECKOUT=$CHECKOUT + VERIFY_XUM_COMMIT=$(git -C "$CHECKOUT" rev-parse HEAD) + AGENT_BROWSER_SESSION="verify-xum-$VERIFY_XUM_RUN_ID" + mkdir -m 700 "$VERIFY_XUM_RUN" + mkdir -p "$VERIFY_XUM_EVIDENCE" "$VERIFY_XUM_RUN/scratch" + # The server tree's TMPDIR is a folder of this run, never the caller's: the sandbox makes its + # XUM_ROOT there, and the backend writes server.lock with the token into it. In a shared TMPDIR + # another account could swap that folder before the backend writes. + mkdir -m 700 "$VERIFY_XUM_RUN/tmp" + require_private_dir "$VERIFY_XUM_RUN/tmp" + log="$VERIFY_XUM_RUN/server.log" + # Milliseconds: a stamp from earlier in the launch's own second is a stale build. + t0=$(date +%s%3N) + + local env_without_live_xum=(env -i PATH="$PATH" HOME="$HOME" USER="${USER:-}" LANG="${LANG:-C.UTF-8}" + TERM=dumb TMPDIR="$VERIFY_XUM_RUN/tmp" XUM_MOCK_AI=1 XUM_DISABLE_TELEMETRY=1 + VERIFY_XUM_SERVER_OF="$VERIFY_XUM_RUN_ID") + # A shell that ran `eval "$(verify-xum.sh env)"` still exports the previous run's values. + VERIFY_XUM_PGID='' VERIFY_XUM_ROOT='' VERIFY_XUM_BACKEND='' VERIFY_XUM_URL='' + VERIFY_XUM_WORKSPACE_ID='' VERIFY_XUM_WORKTREE='' + write_state + ln -sfn "$VERIFY_XUM_RUN" "$CURRENT_RUN_LINK" + trap launch_failed EXIT + # The new group's leader appends its pid, which is the group id, to state.env before it starts + # make. From then on the launch guard and cleanup find the group, wherever the launcher dies. + # The leader carries VERIFY_XUM_SERVER_OF from its start, so group_is_ours matches it as soon + # as the id is written. It holds the launch flock (fd 9) until then and closes it before make. + # (setsid(1) forks when its caller already leads a process group, so $! can miss the group.) + # shellcheck disable=SC2016 + VERIFY_XUM_SERVER_OF="$VERIFY_XUM_RUN_ID" setsid sh -c 'printf "export VERIFY_XUM_PGID=%s\n" "$$" >>"$1"; exec 9>&-; shift; exec "$@"' sh \ + "$VERIFY_XUM_RUN/state.env" "${env_without_live_xum[@]}" \ + make -C "$CHECKOUT" dev-server-sandbox DEV_SERVER_SANDBOX_ARGS="--clean-providers --clean-projects" \ + >"$log" 2>&1 </dev/null & + until VERIFY_XUM_PGID=$(sed -n 's/^export VERIFY_XUM_PGID=\([0-9][0-9]*\)$/\1/p' "$VERIFY_XUM_RUN/state.env" | tail -n 1) + [[ -n $VERIFY_XUM_PGID ]]; do + before_launch_deadline + sleep 0.1 + done + exec 9>&- + write_state + + while true; do + VERIFY_XUM_ROOT=$(sed -n 's/^ XUM_ROOT: *//p' "$log") + backend_port=$(sed -n 's|^ Backend: *http://127.0.0.1:||p' "$log") + vite_port=$(sed -n 's|^ Frontend: *http://localhost:||p' "$log") + [[ -n $VERIFY_XUM_ROOT && -n $backend_port && -n $vite_port ]] && break + group_alive "$VERIFY_XUM_PGID" || die "make dev-server-sandbox exited during startup" + before_launch_deadline + sleep 0.5 + done + # The recipes assume default preferences. With both --clean flags the sandbox copies no config. + grep -q '^ Seed config: *(none)$' "$log" || die "the sandbox copied a config.json, so the run would not start from default preferences" + VERIFY_XUM_BACKEND="http://127.0.0.1:$backend_port" + VERIFY_XUM_URL="http://127.0.0.1:$vite_port" + write_state + + until backend_ready "$t0" && probe_before_launch_deadline "$VERIFY_XUM_URL/"; do + before_launch_deadline + group_alive "$VERIFY_XUM_PGID" || die "the dev server exited before it was ready" + backend_crashed_after_rebuild "$log" && die "the backend crashed after its rebuild (nodemon waits for a file change)" + sleep 1 + done + # Keep the token for redaction (run_token) while server.lock is gone. The run folder is 0700, + # and the file is 0600. + token=$(json_field_or_empty "$VERIFY_XUM_ROOT/server.lock" token) + [[ -z $token ]] || (umask 077 && printf '%s' "$token" >"$VERIFY_XUM_RUN/token") + + seed + write_state + trap - EXIT + cat "$VERIFY_XUM_RUN/state.env" + echo "ready: $VERIFY_XUM_URL/workspace/$VERIFY_XUM_WORKSPACE_ID" +} + +serves_vite_ui() { + # Not `curl | grep -q`: grep exits at the match, and under pipefail curl's SIGPIPE fails the check. + [[ $(curl -fs --max-time 5 "$1/" 2>/dev/null) == *"/@vite/client"* ]] +} + +pid_in_group() { + [[ -n $1 && $(ps -o pgid= -p "$1" 2>/dev/null | tr -d ' ') == "$2" ]] +} + +cmd_doctor() { + load_run "${1:-}" + local failed=0 backend_pid version vite_port vite_pid + check() { + local label=$1 + shift + if "$@"; then + echo "ok $label" + else + echo "FAIL $label" + failed=1 + fi + } + + check "process group $VERIFY_XUM_PGID is alive and carries this run's id" group_is_ours "$VERIFY_XUM_PGID" "$VERIFY_XUM_RUN_ID" + check "this checkout's newest run ($CURRENT_RUN_LINK) is this run" test "$(readlink "$CURRENT_RUN_LINK" 2>/dev/null)" = "$VERIFY_XUM_RUN" + backend_pid=$(json_field_or_empty "$VERIFY_XUM_ROOT/server.lock" pid) + check "backend pid ${backend_pid:-none} (server.lock in $VERIFY_XUM_ROOT) belongs to this run" \ + pid_in_group "$backend_pid" "$VERIFY_XUM_PGID" + check "server.lock names $VERIFY_XUM_BACKEND" \ + test "$(json_field_or_empty "$VERIFY_XUM_ROOT/server.lock" baseUrl)" = "$VERIFY_XUM_BACKEND" + check "GET $VERIFY_XUM_BACKEND/health answers" backend_curl -fs --max-time 5 -o /dev/null "$VERIFY_XUM_BACKEND/health" + version=$(backend_curl -fs --max-time 5 "$VERIFY_XUM_BACKEND/version" 2>/dev/null | json_field_or_empty /dev/stdin git_commit || true) + check "backend build ${version:-unknown} is the launch commit $VERIFY_XUM_COMMIT" \ + test -n "$version" -a "${VERIFY_XUM_COMMIT#"$version"}" != "$VERIFY_XUM_COMMIT" + if [[ $(git -C "$VERIFY_XUM_CHECKOUT" rev-parse HEAD) != "$VERIFY_XUM_COMMIT" ]]; then + echo "note HEAD moved since launch. Source edits hot-reload, /version keeps the launch commit." + fi + check "backend env has XUM_MOCK_AI=1" grep -qszx 'XUM_MOCK_AI=1' "/proc/${backend_pid:-0}/environ" + vite_port=${VERIFY_XUM_URL##*:} + vite_pid=$(ss -Hltnp "sport = :$vite_port" 2>/dev/null | sed -n 's/.*pid=\([0-9]*\).*/\1/p' | head -n 1) + check "app port $vite_port is owned by this run (pid ${vite_pid:-none})" pid_in_group "$vite_pid" "$VERIFY_XUM_PGID" + check "$VERIFY_XUM_URL serves the Vite dev UI" serves_vite_ui "$VERIFY_XUM_URL" + return "$failed" +} + +cmd_env() { + load_run "${1:-}" + cat "$VERIFY_XUM_RUN/state.env" + # The feature recipes call the helper by its bare name. $PATH expands when the caller evals. + # shellcheck disable=SC2016 + printf 'export PATH=%q:"$PATH"\n' "$SCRIPT_DIR" + echo "unset $GIT_REPO_ENV" +} + +# Opens PATH of the run's UI in the run's browser session. When the backend has an auth token, the +# URL carries `?token=` (the app keeps the token and strips it from the address bar). node reads +# the token from server.lock and pipes the command to `agent-browser batch` on stdin, so the token +# is in no process's argv or environment: every local account can read argv. +cmd_open() { + local path=${1:-} out + [[ $path == /* ]] || die "open PATH must start with / (for example /workspace/<id>)" + load_run "${2:-}" + # Checked before agent-browser runs. A PATH like //host, /\host or /<tab>/host resolves to + # another origin: never send the token there. One that normalizes to //... (/..//host) stays on + # the origin, but the app cannot strip ?token= from such an address, so the token stays in it. + node -e ' + const [base, path] = process.argv.slice(1); + const url = new URL(path, base); + if (url.origin !== new URL(base).origin || url.pathname.startsWith("//")) { + process.stderr.write("verify-xum: open PATH must stay on " + base + " and must not start with // once normalized\n"); + process.exit(1); + }' "$VERIFY_XUM_URL" "$path" || exit 1 + # agent-browser serves each session's screen and input as a WebSocket on 127.0.0.1 that every + # local account can connect to, and a new daemon starts it on its own (no setting keeps it + # off). Turn it off before the token reaches the browser, on every open because a restarted + # daemon starts it again. `stream disable` also drops connected clients, and starts the daemon + # when none runs: the browser it launches later keeps the stream off. + agent-browser --session "$AGENT_BROWSER_SESSION" stream disable >/dev/null 2>&1 || true + stream_is_off || die "agent-browser's stream for session $AGENT_BROWSER_SESSION is still on. Not opening the page with the token." + # shellcheck disable=SC2001 # sed's character class; bash patterns have none + out=$(node -e ' + const fs = require("fs"); + const [serverLock, base, path] = process.argv.slice(1); + let token = ""; + try { + token = JSON.parse(fs.readFileSync(serverLock, "utf8")).token ?? ""; + } catch {} + const url = new URL(path, base); + if (token) url.searchParams.set("token", token); + process.stdout.write(JSON.stringify([["open", url.href]]));' \ + "$VERIFY_XUM_ROOT/server.lock" "$VERIFY_XUM_URL" "$path" \ + | agent-browser --session "$AGENT_BROWSER_SESSION" batch --bail 2>&1) \ + || die "agent-browser could not open $path: $(sed 's/token=[^&[:space:]]*/token=<redacted>/g' <<<"$out")" + # The success output repeats the URL with the token: it is not printed. + if ! stream_is_off; then + agent-browser --session "$AGENT_BROWSER_SESSION" close >/dev/null 2>&1 || true + die "agent-browser's stream for session $AGENT_BROWSER_SESSION came on during the open. Closed the session." + fi +} + +# True when the session's agent-browser stream (see cmd_open) is off. +stream_is_off() { + [[ $(agent-browser --session "$AGENT_BROWSER_SESSION" stream status --json 2>/dev/null \ + | node -e 'try { console.log(JSON.parse(require("fs").readFileSync(0, "utf8")).data.enabled === false) } catch { console.log(false) }') == true ]] +} + +# Prints FIELD (id, archivedAt, ...) of the workspace named NAME in the run's config.json, or +# nothing when that field is unset. Exits 1 unless exactly one workspace has that name. The +# recipes read ids with it: the app makes them up at create time. +cmd_workspace_field() { + local name=${1:-} field=${2:-} + [[ -n $name && -n $field ]] || die "usage: verify-xum.sh workspace-field NAME FIELD [RUN]" + load_run "${3:-}" + node -e ' + const fs = require("fs"); + const [configPath, name, field] = process.argv.slice(1); + const config = JSON.parse(fs.readFileSync(configPath, "utf8")); + const matches = config.projects.flatMap(([, project]) => project.workspaces ?? []).filter((w) => w.name === name); + if (matches.length !== 1) { + process.stderr.write("verify-xum: " + matches.length + " workspaces are named " + name + " in " + configPath + "\n"); + process.exit(1); + } + const value = matches[0][field]; + if (value !== undefined && value !== null) console.log(String(value));' \ + "$VERIFY_XUM_ROOT/config.json" "$name" "$field" +} + +# The backend's token: server.lock's, else the copy launch keeps in the private run folder. A +# restarting backend deletes server.lock for a moment, and a stopped one for good. +run_token() { + local token + token=$(json_field_or_empty "$VERIFY_XUM_ROOT/server.lock" token) + [[ -n $token || ! -f $VERIFY_XUM_RUN/token ]] || token=$(<"$VERIFY_XUM_RUN/token") + printf '%s' "$token" +} + +# Evidence is meant to be shared, so no text in it may keep the backend's token: the page keeps +# ?token= in its URL when the app fails before it strips it. This replaces the token (raw, and +# URL-encoded as a path part and as a form value) and the value of any token= query parameter +# with <redacted>. With no PATH it filters stdin to stdout. With PATHs it rewrites each text file +# under them in place, skipping binary files and symlinks. The token is $REDACT_TOKEN when set +# (cleanup reads it before the sandbox deletes server.lock), else server.lock's. node gets it on +# fd 3, never in argv. +REDACT_TOKEN= +redact_token() { # redact_token [PATH...] + local token=$REDACT_TOKEN + [[ -n $token ]] || token=$(run_token) + # shellcheck disable=SC2016 # $1 is the regex group, for node + node -e ' + const fs = require("fs"); + const path = require("path"); + const token = fs.readFileSync(3, "utf8"); + const forms = [...new Set([token, encodeURIComponent(token), new URLSearchParams({ t: token }).toString().slice(2)])].filter(Boolean); + const redact = (text) => { + for (const form of forms) text = text.split(form).join("<redacted>"); + // A value of at least one character: an empty ?token= and a placeholder such as + // ?token=<token> (the Makefile banner in server.log) stay as they are. + return text.replace(/([?&]token=)[^&#\s"\x27<>]+/g, "$1<redacted>"); + }; + const filesUnder = (p) => { + const st = fs.lstatSync(p); + if (st.isDirectory()) return fs.readdirSync(p).flatMap((name) => filesUnder(path.join(p, name))); + return st.isFile() ? [p] : []; + }; + const paths = process.argv.slice(1); + if (paths.length === 0) { + process.stdout.write(redact(fs.readFileSync(0, "utf8"))); + process.exit(0); + } + for (const file of paths.filter((p) => fs.existsSync(p)).flatMap(filesUnder)) { + const data = fs.readFileSync(file); + if (data.subarray(0, 8192).includes(0)) continue; + const text = data.toString("utf8"); + const clean = redact(text); + if (clean !== text) fs.writeFileSync(file, clean); + }' "$@" 3< <(printf %s "$token") +} + +cmd_capture() { + local name=${1:-} base + [[ $name =~ ^[a-z0-9][a-z0-9._-]*(/[a-z0-9][a-z0-9._-]*)*$ ]] || die "capture NAME must look like <feature>/<step>, lowercase" + load_run "${2:-}" + base="$VERIFY_XUM_EVIDENCE/$name" + mkdir -p "$(dirname "$base")" + agent-browser --session "$AGENT_BROWSER_SESSION" get url | redact_token >"$base.url.txt" + agent-browser --session "$AGENT_BROWSER_SESSION" snapshot | redact_token >"$base.aria.txt" + agent-browser --session "$AGENT_BROWSER_SESSION" screenshot "$base.png" >/dev/null + echo "$base.png $base.aria.txt $base.url.txt" +} + +CHUNK_LOAD_FAILED_JS="[...document.querySelectorAll('[role=\"alert\"]')].some((e) => e.textContent.includes(' failed to load.'))" +SETTLED_OR_CHUNK_FAILED_JS="(() => { + if ($CHUNK_LOAD_FAILED_JS) return true; + const log = document.querySelector('[role=\"log\"][aria-label=\"Conversation transcript\"]'); + if (location.pathname.startsWith('/workspace/') && log?.getAttribute('aria-busy') !== 'false') return false; + return !document.getAnimations().some((a) => a instanceof CSSTransition); +})()" + +cmd_idle() { + load_run "${1:-}" + agent-browser --session "$AGENT_BROWSER_SESSION" wait --fn "$SETTLED_OR_CHUNK_FAILED_JS" >/dev/null + if [[ $(agent-browser --session "$AGENT_BROWSER_SESSION" eval "$CHUNK_LOAD_FAILED_JS") == true ]]; then + echo "verify-xum: the page shows '<name> failed to load.': a module fetch from Vite failed. Run agent-browser reload, then repeat the step." >&2 + exit 3 + fi +} + +cmd_send_ready() { + load_run "${1:-}" + agent-browser --session "$AGENT_BROWSER_SESSION" wait --fn \ + "document.querySelector('[aria-label=\"Send message\"]')?.disabled === false" >/dev/null +} + +is_dev_server_sandbox_root() { + [[ -n $1 && -d $1 && $(basename "$1") == mux-dev-server-* ]] +} + +cmd_cleanup() { + load_run "${1:-}" + local i + # Read the token before the group stops: the backend then deletes server.lock. + REDACT_TOKEN=$(run_token) + # agent-browser `close` on a session that does not exist starts that session first. + if [[ $(agent-browser session list 2>/dev/null) == *"$AGENT_BROWSER_SESSION"* ]]; then + agent-browser --session "$AGENT_BROWSER_SESSION" close >/dev/null 2>&1 || true + fi + if group_is_ours "$VERIFY_XUM_PGID" "$VERIFY_XUM_RUN_ID"; then + kill -TERM -- "-$VERIFY_XUM_PGID" 2>/dev/null || true + for ((i = 0; i < 20; i++)); do + group_alive "$VERIFY_XUM_PGID" || break + sleep 1 + done + if group_is_ours "$VERIFY_XUM_PGID" "$VERIFY_XUM_RUN_ID"; then + kill -KILL -- "-$VERIFY_XUM_PGID" 2>/dev/null || true + fi + elif group_alive "$VERIFY_XUM_PGID"; then + echo "verify-xum: process group $VERIFY_XUM_PGID is not this run's dev server. Not signaling it." >&2 + fi + # Text you saved by hand is evidence too. + redact_token "$VERIFY_XUM_EVIDENCE" "$VERIFY_XUM_RUN/server.log" || true + if is_dev_server_sandbox_root "$VERIFY_XUM_ROOT"; then + rm -rf "$VERIFY_XUM_ROOT" + fi + rm -rf "$VERIFY_XUM_RUN/scratch" "$VERIFY_XUM_RUN/tmp" + [[ $(readlink "$CURRENT_RUN_LINK" 2>/dev/null) == "$VERIFY_XUM_RUN" ]] && rm -f "$CURRENT_RUN_LINK" + VERIFY_XUM_PGID= + write_state + # Last: the redaction above used it. + rm -f "$VERIFY_XUM_RUN/token" + echo "cleaned up run $VERIFY_XUM_RUN_ID; evidence kept in $VERIFY_XUM_EVIDENCE ($(find "$VERIFY_XUM_EVIDENCE" -type f | wc -l | tr -d ' ') files) and $VERIFY_XUM_RUN/server.log" +} + +case ${1:-} in + launch) cmd_launch ;; + doctor) cmd_doctor "${2:-}" ;; + env) cmd_env "${2:-}" ;; + open) cmd_open "${2:-}" "${3:-}" ;; + workspace-field) cmd_workspace_field "${2:-}" "${3:-}" "${4:-}" ;; + capture) cmd_capture "${2:-}" "${3:-}" ;; + idle) cmd_idle "${2:-}" ;; + send-ready) cmd_send_ready "${2:-}" ;; + cleanup) cmd_cleanup "${2:-}" ;; + *) die "usage: verify-xum.sh launch | doctor [RUN] | env [RUN] | open PATH [RUN] | workspace-field NAME FIELD [RUN] | capture NAME [RUN] | idle [RUN] | send-ready [RUN] | cleanup [RUN]" ;; +esac