Repository navigation
🤖 feat: add verify-xum skill that drives the real Xum web UI - #7895
Conversation
A project-local verification skill: verify-xum.sh launches one mock-AI make dev-server-sandbox per checkout (env -i, setsid process group, seeded project and workspace), checks it with doctor, waits with idle and send-ready, captures evidence that survives cleanup, and a feature map covers workspaces, chat, review, command palette and settings.
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (2)
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae3d23714a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… bounded probes, group identity - The newest-run link lives in the checkout's git dir next to the lock, so two checkouts that share VERIFY_XUM_HOME never load each other's run. - launch takes the checkout lock under flock, so two launches at once cannot both start. - Every curl has --max-time, and the readiness loop uses a wall-clock deadline. - doctor deletes the previous version.json before it asks /version again. - cleanup signals the group only while a process in it has VERIFY_XUM_SERVER_OF=<run id>. - SKILL.md says a canned todo_write still writes its TODO list.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8601eba987
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ound every launch phase - settings.md and command-palette.md wait for the option to be selected before Enter. The first palette open after a page load shows a loading input that drops Enter (LazyCommandPalette.tsx:37/60), so an Enter typed right after the text was lost. - settings.md waits until no modal dialog is open after Escape. The palette shortcuts do nothing while one is (isDialogOpen(), keybinds.ts:291, App.tsx:1133). - README and SKILL.md: run a step's commands as one && chain, so a failed wait stops it instead of typing into the composer. - env also puts the helper on PATH, so the recipes' bare verify-xum.sh resolves. - launch checks its deadline during port discovery, the readiness loop, every seed API call and the workspace-setup wait.
…unnable review recipe - load_run refuses a run that another checkout launched, so cleanup and doctor never act on it. - cleanup signals a group only while its live leader is still the launch's make dev-server-sandbox, and blanks the group id in state.env after the stop. - doctor reads /version without writing a file. SKILL.md says every check line reads ok. - review.md Restore unmarks the hunk, unchecks Uncommitted and closes the Review tab, so the recipe runs again in the same run. - settings.md drives Ctrl+, too. chat.md and workspaces.md create the evidence folder first. - SKILL.md: Linux only, the xum run snippet runs under env -i with a fake key (it refuses without one, and the old snippet used the caller's real keys), two more proof standards. - workspaces.md records that document.hasFocus() was true on agent-browser 0.33.2.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ddacc6f52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…absolute The live Xum's server URL and token are also in the env as legacy MUX_* names, which the CLI reads, so env -u of the XUM_* names alone did not keep them out. A relative VERIFY_XUM_HOME made the run link in the git dir point at a path relative to the git dir.
… exited, end recipes on the workspace - workspaces.md runs idle after each create's reply. The reply text shows before the stream ends, and archiving a streaming workspace opens a confirm dialog instead. - cleanup treats a zombie group leader (empty cmdline) like an exited one, so it still stops the group's live members. - review.md says to repeat from Open the sidebar after a cold-cache page reload. - settings.md Restore re-opens the workspace: Escape from a Settings URL lands on /project.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90251e3cbf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ive TMPDIR absolute - README: start every recipe whose preconditions name Verify playground by opening it. The workspaces and command-palette recipes end on release-notes, and the recipes read their evidence from $VERIFY_XUM_WORKSPACE_ID. - make -C runs the sandbox from the checkout, so a relative TMPDIR resolved there while the script resolved the logged XUM_ROOT against the caller's directory.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d27855043
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Stopping the review loop at round 5 of 6 on head
This PR changes only files under Generated with |
… saved chat reply - launch keeps the flock fd open, and the server's make and dev-server-sandbox.ts inherit it, so the lock stays held while the server runs at whatever point the launcher dies. A second launch is refused at once. launch saves state, the run link and its cleanup trap before the spawn, and cleanup reads the group id and XUM_ROOT from the pidfile and server log when a killed launch never saved them. - chat.md waits until chat.jsonl has the second reply before it reads the history. The transcript shows the reply before the server saves it at stream end.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4c7f314be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review-cap waiver: ThomasK33 approved one seventh review round for this PR (option B), recorded before the round starts. Scope of round 7:
If round 7 raises a new valid P1 or P2, the loop stops and reports instead of iterating. Generated with |
…rited lock - launch refuses while the process group recorded for this checkout's newest run is alive, checked with the same leader-cmdline and run-id test that cleanup uses. The group's leader appends its id to the run's state.env before it starts make, so the guard and cleanup find it at whatever point the launcher dies. The flock now only serializes the check and the spawn, and the server does not inherit it. - Removed the inherited lock fd, the separate pidfile and lock file, and cleanup's pidfile and server-log fallbacks. state.env is written atomically. - doctor checks that the checkout's newest-run link names the run. - review.md: after a cold-cache reload the sidebar and Review tab stay open, so the recovery continues at Include uncommitted edits. - command-palette.md: #7897 is on main, so the loading input keeps an early Enter. The wait for the selected option stays.
…deletes its root when the group stops
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a450f19ed0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review-cap waiver: ThomasK33 approved one eighth review round for this PR (option A), recorded before the round starts. Scope of round 8:
If round 8 raises a new valid P1 or P2, the loop stops and reports. Generated with |
When the app fails before it strips ?token= (a renderer error, a chunk that does not load), the browser stays on a URL with the token, and capture saved that URL into the evidence, which is meant to be shared. capture now passes the URL and the snapshot through redact_token, which replaces the token (raw and URL-encoded) and any token= value with <redacted>. Cleanup does the same in every text file of the evidence and in the server log, before it deletes server.lock. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5caa4030d6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ing the server Stopping the group makes the backend delete server.lock within about a second, so cleanup's redaction loaded an empty token and left a bare token in the evidence. Cleanup now reads the token first, and the redactor gets it on fd 3. The token= pattern also matches only a non-empty value, so an empty ?token= and the Makefile banner's ?token=<token> stay as they are. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcfed1dbd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…eaches the browser - agent-browser 0.33 serves each session's screen and input as a WebSocket on 127.0.0.1 that any local account can connect to, and a new daemon starts it on its own. open now runs stream disable every time, refuses to open while stream status reports it on, and closes the session if it comes on during the open. stream disable also drops clients that connected before. - SKILL.md says to run the skill only where you are the only local account: Chrome's DevTools port on 127.0.0.1 asks for no credentials, and agent-browser cannot turn it off. - Launch keeps a copy of the token in the private run folder (0600), so capture and cleanup can redact while server.lock is gone (a backend restart). Cleanup deletes the copy last. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed843b41ed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed843b41ed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A run home inside the checkout publishes the token cache and evidence through Vite. File modes do not help because Vite runs as the same user. Launch now refuses that home before creating any folder. Every command that loads a run checks it too, including a run an older version left there. The check resolves symlinks and .. physically and covers both the current checkout and the main worktree of a linked checkout. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh` • Cost: `$153.98`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=xhigh costs=153.98 -->
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f02d0b04fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Vite allows its detected workspace root (server.fs.allow defaults to searchForWorkspaceRoot(root)), and that root can be a parent of the checkout, for example a folder whose package.json declares workspaces. A run home under that parent passed the checkout check, and Vite served <run>/token to an unauthenticated request. Every home check now asks the checkout's own vite for that root (searchForWorkspaceRoot of the physical checkout, as vite started there computes it) and refuses a home inside it, in addition to the checkout and the main worktree. When vite cannot answer, the check refuses. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh` • Cost: `$166.57`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=xhigh costs=166.57 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83da137e2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
CI's shellcheck (0.8.0) reports SC2015 on `A && B || die`. The check is now an if/else with the same result: vite_workspace_root prints the root only when the checkout's vite reports one absolute path, and refuses otherwise. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh` • Cost: `$171.54`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=xhigh costs=171.54 -->
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Why
Agents that change Xum have no scripted way to drive the real app and prove the change the way a user sees it. This adds a project-local
verify-xumskill: one command starts an isolated, mock-AI dev server from the checkout, and a feature map says how to drive and check each main feature, with evidence that survives teardown.What changed
.xum/skills/verify-xum/scripts/verify-xum.shhaslaunch,doctor,env,capture,idle,send-readyandcleanup. Launch wrapsmake dev-server-sandbox --clean-providers --clean-projectswithXUM_MOCK_AI=1, anenv -iallowlist and its own process group, and seeds a trusted demo project and one workspace through the HTTP API.SKILL.mdcovers launch, doctor, drive, evidence, cleanup and the traps found while proving it.features/maps five features (workspaces, chat, review, command palette, settings), each with exactagent-browsercommands, the expected UI state and an on-disk side effect to check.Scope
dev-desktop-sandbox,electronandbug-bashskills, which the skill links to.(hover: none), no window focus), the mock AI, or the Vite dev server.Blast Radius
New files only, under
.xum/skills/. The script never touches~/.xum: it refuses a second server per checkout, kills only the process group it started, and deletes only themux-dev-server-*root that the sandbox created.Verification
SKILL.mdand every step of the five recipes on the final head, with an assertion per step: 108 checks passed, 0 failed, about 2 minutes, no model calls. Aftercleanup, the sandbox root, the process group, the checkout lock and the browser session were gone, and the 46 evidence files remained.launch, the wrong Bun on PATH, a launch that times out (it cleans up), anddoctoron a stopped run (9 FAIL lines, exit 1).make static-checkpassed, andverify-xumshows up as a project skill.The Review tab after a worktree edit, and a stopped mock stream:
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh