From 1790e313de94d8ba1e98a5fefa998b6cc3d0e9d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Oskar=20Kwas=CC=81niewski?= Date: Fri, 6 Nov 2026 22:23:37 +0100 Subject: [PATCH 1/2] docs(skills): bug bash repros from the finding, charters sized to the clock Repro tests start as agent.act steps plus a closing agent.assert, with no locators; scripted hardening over MCP is opt-in. Each verifier runs its own repros into its own output and the report runs them all once. Confirmed means failing on the reported defect, read from the message for an act. Charters get ~15 min (was 6 steps), a time-limited bash keeps running new charters and reports each bug once confirmed, and short seeded passwords are called out. --- .changeset/skill-bug-bash-repros.md | 5 + docs/bug-bash.mdx | 11 ++- docs/reference/mcp.mdx | 4 +- .../skills/create-verification-skill/SKILL.md | 2 +- .../verify-playground/SKILL.example.md | 4 +- plugins/e2e/skills/e2e/references/bug-bash.md | 98 +++++++++++++------ skills/create-verification-skill/SKILL.md | 2 +- .../verify-playground/SKILL.example.md | 4 +- skills/e2e/references/bug-bash.md | 98 +++++++++++++------ 9 files changed, 159 insertions(+), 69 deletions(-) create mode 100644 .changeset/skill-bug-bash-repros.md diff --git a/.changeset/skill-bug-bash-repros.md b/.changeset/skill-bug-bash-repros.md new file mode 100644 index 0000000000..7155402ee8 --- /dev/null +++ b/.changeset/skill-bug-bash-repros.md @@ -0,0 +1,5 @@ +--- +"e2e": patch +--- + +The bundled e2e skill's bug bash writes each repro test from the finding as `agent.act` steps and a closing `agent.assert`, so confirming a bug needs no locators. Each verifier runs its own repros into its own output directory, and the report runs them all once with `--tag bugbash`. A finding is confirmed when its repro fails on the reported defect, and the skill says how to read a failed `act`. Charters get about fifteen minutes each, a bash under a time limit keeps running new charters until near the end and reports each bug as its repro confirms it, and an account with a password under 6 characters is called out before the config fails. diff --git a/docs/bug-bash.mdx b/docs/bug-bash.mdx index 9f980b7947..f50b489cdd 100644 --- a/docs/bug-bash.mdx +++ b/docs/bug-bash.mdx @@ -75,11 +75,12 @@ Bug bash the checkout and account settings on this branch. 5. Rejects findings that have a known cause. The cause can be a limit of the explorer (for example, a link that opens a new tab), the local environment, the intended design, or the seed data. -6. Writes a repro test that asserts the expected behavior for each - remaining finding. It can start one subagent per area, each with its own - [`e2e mcp` session](/mcp-server) to check locators. -7. Runs each repro test. The finding is a bug only when the test fails with - `ASSERTION_FAILED`. +6. Writes a repro test for each remaining finding: the finding's steps as + `agent.act` calls and its expected behavior as the closing + `agent.assert`. It can start one subagent per area. +7. Runs the repro tests. The finding is a bug only when its test fails on + the reported defect: `ASSERTION_FAILED` from the closing check, or an + `act` that fails on the defect itself. 8. Reports, in this order: confirmed bugs, production risks that it cannot verify locally, rejected findings by reason, and warnings. diff --git a/docs/reference/mcp.mdx b/docs/reference/mcp.mdx index c85fcbc9bc..747880a832 100644 --- a/docs/reference/mcp.mdx +++ b/docs/reference/mcp.mdx @@ -652,8 +652,8 @@ Several sessions can be open at once, up to `--max-sessions`, each on its own attempt and its own browser or device. Parallel agents, such as a coding agent's subagents, share one server this way: each opens a session and passes its id to every `tools`, `call`, and `close_session`. -[Bug bashes](/bug-bash) use this: every verifying subagent drives its own -session. +[Bug bashes](/bug-bash) use this: a verifying subagent that hardens a repro +into a scripted test drives its own session. | Throws | When | | --- | --- | diff --git a/plugins/e2e/skills/create-verification-skill/SKILL.md b/plugins/e2e/skills/create-verification-skill/SKILL.md index 9ab467b358..e40d91bbe2 100644 --- a/plugins/e2e/skills/create-verification-skill/SKILL.md +++ b/plugins/e2e/skills/create-verification-skill/SKILL.md @@ -71,7 +71,7 @@ found, no placeholders. | Doctor | One read-only check that answers "is this instance worth driving?". `npx e2e list` loads the config and prints every test-target pair without starting anything, so a config error shows here first. Then the smoke test, a model-free test that opens the app and pins one landmark; exit `0` means the app starts, the engine runs, and the landmark renders. No smoke test in the project: write `tests/smoke.e2e.ts` now. A signed-in app's auth doctor is an ordinary signed-in smoke test that consumes a named session: selecting that test runs its setup and checks the restored state, so one command proves the app starts, the session restores, and a signed-in landmark renders. Running the setup file alone selects no test and exits `NO_TESTS`, so it is not the doctor. An agent runs Doctor first whenever anything looks off | | Drive | Two lanes and when to use each. Live over MCP: `open_session`, `observe`, the verbs, and `locate` before any locator is written (topic `mcp`); list the project tools and credentials the session's catalog offers. Scripted: a test under the `tests` glob, `screen` where the name on screen is known, `agent.act` where the flow varies (topic `writing-tests`). Prefer stable handles, roles and accessible names, labels, test ids, routes. Name the credentials and sessions a signed-in drive starts from | | Evidence | What a proof keeps and where. Every run writes `.e2e/report.json`; a failed test gets `.e2e/results//trace.md`, `--trace on` gives one to passing tests too, `--video` records the attempt (topic `running`), `app.screenshot('')` saves a named frame (topic `writing-tests`). The next run clears `.e2e/results/`, Doctor included, so a proof that must outlive it runs with `--output .e2e/proof/` or its attempt directory is copied out before anything else runs. State the proof standards (below) | -| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/.e2e.ts` fails with `ASSERTION_FAILED` on the assertion that encodes it | +| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/.e2e.ts` fails on the reported defect, as topic `bug-bash` step 6 reads it | | Cleanup | What to tear down. The runner stops the app it started and `close_session` stops the one a session started; a server the user started stays theirs. Never kill by process name; stop what you started. Cleanup removes instances and scratch state (seeded accounts, scaffolding, charters whose findings were all rejected), never the evidence; name where the evidence is | | Helpers | Any script the skill ships is executable and its invocation is in the skill body. A helper the reader has to reverse-engineer is not a helper. Most projects need none; the e2e CLI is the helper | diff --git a/plugins/e2e/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md b/plugins/e2e/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md index 69c29b55ce..ed41110eab 100644 --- a/plugins/e2e/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md +++ b/plugins/e2e/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md @@ -70,8 +70,8 @@ fan-out in topic `bug-bash` step 3 reads. `.e2e/bugbash/` is in `.gitignore`, with `.e2e/proof/`. One explorer at a time needs no isolation change; several need `pnpm app` started first and `reuseExisting: true` on the command, since the port is fixed. A finding is a bug once -`tests/bugbash/.e2e.ts` fails with `ASSERTION_FAILED` on the -assertion that encodes it; anything else is the explorer's claim. +`tests/bugbash/.e2e.ts` fails on the reported defect, as topic +`bug-bash` step 6 reads it; anything else is the explorer's claim. ## Cleanup diff --git a/plugins/e2e/skills/e2e/references/bug-bash.md b/plugins/e2e/skills/e2e/references/bug-bash.md index a75eacd82d..1bc48ddd26 100644 --- a/plugins/e2e/skills/e2e/references/bug-bash.md +++ b/plugins/e2e/skills/e2e/references/bug-bash.md @@ -103,8 +103,11 @@ export default { ``` Set `BUGBASH_PASSWORD` to 6 or more characters first, or the config fails to -load. Pass `--config e2e.bugbash.config.ts` to every command below, and -`open_session {config: "e2e.bugbash.config.ts"}` over MCP. +load: a shorter password can't be redacted without rewriting ordinary text, +so an account the app seeds with one (`admin`/`admin`) needs a longer +password before it can be a credential. Pass `--config e2e.bugbash.config.ts` +to every command below, and `open_session {config: "e2e.bugbash.config.ts"}` +over MCP. ## 2. Plan charters @@ -151,9 +154,14 @@ CHARTERS while IFS='|' read -r slug target agent charter; do [ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter" done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \ - 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 6 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ + 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 15 --timeout 900000 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ ``` +`--timeout` ends a charter with its assessment whatever its step count, and +a step takes about a minute, so the flags above give each charter about +fifteen minutes. Under a time limit, size the first round to end by about a +third of it. + The log's summary prints `AI` (cost) and `Duration`. Exit code `1` means issues were reported: read the log either way. Exit codes `2` and `3` are setup and environment problems; fix them and rerun that charter alone. @@ -188,40 +196,70 @@ Sort every finding before writing any test. Read the source to sort them. A finding is a model's claim; prove each candidate before reporting it. When your client can start subagents (Claude Code's Agent or Task tool, for one), start one per area with its three to five candidates as the log printed -them and these steps, up to four at a time, the `e2e mcp` server's default -session limit. Each reports back, per finding: confirmed or rejected, the -root cause as `file:line` when it may read the source, the repro test path, -and the failure it saw. Without subagents, verify one area after another. +them and these steps, up to four at a time (the `e2e mcp` server's default +session limit, which step 4 uses). Each reports back, per finding: +confirmed or rejected, the root cause as `file:line` when it may read the +source, the repro test path, and the failure it saw. Without subagents, +verify one area after another. 1. Read `actual` against the screenshot, or the video when there is none. A finding the evidence contradicts is rejected here. Settle the artifact bucket first: a "dead" link's destination must actually open (a valid href with a prevented default is still dead), a copy claim must show in the rendered screenshot. -2. Write a repro test that follows the reproduction and asserts the - expected behavior, so it fails today and passes once the bug is fixed. - Put it under `bugbash/` inside the directory the config's `tests` glob - covers (`tests/bugbash/.e2e.ts` for the default): a file the glob - does not match is never selected, whatever path you pass to `e2e run`. - Build on the project's fixtures (a `test.extend` fixture, a setup test's - session). Prefer `screen` actions and `expect` with exact values; - `agent.act` for a step that varies, `agent.assert` for an outcome only - judgment can check (topic `writing-tests`). Tag it `{ tags: ['bugbash'] }`. -3. Get exact locators from the live app: with the `e2e mcp` server - registered, `open_session` with the bug-bash config, pass its session id - to every call, walk the reproduction, and `locate` each locator before - writing it (topic `mcp`). Close your session when done. -4. Run the file alone: `npx e2e run tests/bugbash/.e2e.ts`. Confirmed - only when it fails with `ASSERTION_FAILED` on the assertion that encodes - the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, a setup - error) means the test is wrong: fix it and rerun. A passing test means - the bug did not reproduce: reject the finding, say so, and move the test - out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as - a regression test). Only failing repro tests stay there. +2. Write a repro test from the finding: its reproduction as `agent.act` + steps and its expected behavior as the closing `agent.assert`, so it + fails today and passes once the bug is fixed. It needs no locators. Sign + in as the charter did, with its credential as step parameters or a setup + test's session. Assert only the defect: a count or a record another repro + changes makes it fail for the wrong reason. Put it under `bugbash/` + inside the directory the config's `tests` glob covers + (`tests/bugbash/.e2e.ts` for the default): a file the glob does not + match is never selected, whatever path you pass to `e2e run`. Tag it + `{ tags: ['bugbash'] }`. + + ```ts + // tests/bugbash/cart-line-quantity.e2e.ts + import { test } from '@e2e-dev/web'; + import { credentials } from 'e2e'; + + test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => { + const shopper = credentials.user('bb-cart'); + await app.open('/'); + await agent.act('sign in with the given credentials', { + params: { username: shopper.username, password: shopper.password }, + }); + await agent.assert('the app shows you are signed in'); + await agent.act('add "Dune" to the cart'); + await agent.act('set the quantity of "Dune" in the cart to 2'); + await agent.assert('the cart line for "Dune" totals twice its unit price'); + }); + ``` +3. Run your repros with an output directory of your own, `npx e2e run + tests/bugbash/.e2e.ts --output .e2e/bugbash/verify-` (a run + clears its output's results, and another subagent may be reading them), + then one file alone while you fix it. Confirmed when it fails on the + reported defect: `ASSERTION_FAILED` from the closing `assert`, or a + failed `act` whose message names the defect. A failed `act` carries the + code the model chose, so only its message and screenshot tell the defect + apart from a failure short of it. A blocked `act`, a timeout, a setup + error, or a failure that names something else means the repro is wrong: + sharpen the step and rerun. A passing test means the bug did not + reproduce: reject the finding, say so, and move the test out of + `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a + regression test). Only failing repro tests stay there. +4. When the user wants a scripted regression test, harden a confirmed repro + with `screen` actions and exact `expect` values (topic `writing-tests`): + with the `e2e mcp` server registered, `open_session` with the bug-bash + config, pass its session id to every call, walk the reproduction, and + `locate` each locator before writing it (topic `mcp`). Close your session + when done. A scripted step that opens a native dialog needs + `browser.onDialog` first. ## 7. Report -Lead with the confirmed bugs, most severe first. For each: title, path, one +Run every repro once, `npx e2e run --tag bugbash`, so each bug you report +still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one line of expected against actual, root cause when known, steps, screenshot and video paths, repro test, and the charters that found it. Then the environment candidates that are production risks, marked unverified. Then @@ -239,6 +277,10 @@ regression test with its `bugbash` tag removed. - Never report an unverified finding as a bug. "The explorer reported" is not "confirmed". +- Under a time limit, spend it: once a round is verified, run charters for + the areas and postures no charter reached, until about a tenth of the + time is left. Report each bug as soon as its repro confirms it, so a bash + cut off at the limit keeps it. - One charter, one area. A charter that spans the whole app ends at its step budget having skimmed everything. - Seeded data and test accounts the app ships for development are not diff --git a/skills/create-verification-skill/SKILL.md b/skills/create-verification-skill/SKILL.md index 9ab467b358..e40d91bbe2 100644 --- a/skills/create-verification-skill/SKILL.md +++ b/skills/create-verification-skill/SKILL.md @@ -71,7 +71,7 @@ found, no placeholders. | Doctor | One read-only check that answers "is this instance worth driving?". `npx e2e list` loads the config and prints every test-target pair without starting anything, so a config error shows here first. Then the smoke test, a model-free test that opens the app and pins one landmark; exit `0` means the app starts, the engine runs, and the landmark renders. No smoke test in the project: write `tests/smoke.e2e.ts` now. A signed-in app's auth doctor is an ordinary signed-in smoke test that consumes a named session: selecting that test runs its setup and checks the restored state, so one command proves the app starts, the session restores, and a signed-in landmark renders. Running the setup file alone selects no test and exits `NO_TESTS`, so it is not the doctor. An agent runs Doctor first whenever anything looks off | | Drive | Two lanes and when to use each. Live over MCP: `open_session`, `observe`, the verbs, and `locate` before any locator is written (topic `mcp`); list the project tools and credentials the session's catalog offers. Scripted: a test under the `tests` glob, `screen` where the name on screen is known, `agent.act` where the flow varies (topic `writing-tests`). Prefer stable handles, roles and accessible names, labels, test ids, routes. Name the credentials and sessions a signed-in drive starts from | | Evidence | What a proof keeps and where. Every run writes `.e2e/report.json`; a failed test gets `.e2e/results//trace.md`, `--trace on` gives one to passing tests too, `--video` records the attempt (topic `running`), `app.screenshot('')` saves a named frame (topic `writing-tests`). The next run clears `.e2e/results/`, Doctor included, so a proof that must outlive it runs with `--output .e2e/proof/` or its attempt directory is copied out before anything else runs. State the proof standards (below) | -| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/.e2e.ts` fails with `ASSERTION_FAILED` on the assertion that encodes it | +| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/.e2e.ts` fails on the reported defect, as topic `bug-bash` step 6 reads it | | Cleanup | What to tear down. The runner stops the app it started and `close_session` stops the one a session started; a server the user started stays theirs. Never kill by process name; stop what you started. Cleanup removes instances and scratch state (seeded accounts, scaffolding, charters whose findings were all rejected), never the evidence; name where the evidence is | | Helpers | Any script the skill ships is executable and its invocation is in the skill body. A helper the reader has to reverse-engineer is not a helper. Most projects need none; the e2e CLI is the helper | diff --git a/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md b/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md index 69c29b55ce..ed41110eab 100644 --- a/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md +++ b/skills/create-verification-skill/references/example/verify-playground/SKILL.example.md @@ -70,8 +70,8 @@ fan-out in topic `bug-bash` step 3 reads. `.e2e/bugbash/` is in `.gitignore`, with `.e2e/proof/`. One explorer at a time needs no isolation change; several need `pnpm app` started first and `reuseExisting: true` on the command, since the port is fixed. A finding is a bug once -`tests/bugbash/.e2e.ts` fails with `ASSERTION_FAILED` on the -assertion that encodes it; anything else is the explorer's claim. +`tests/bugbash/.e2e.ts` fails on the reported defect, as topic +`bug-bash` step 6 reads it; anything else is the explorer's claim. ## Cleanup diff --git a/skills/e2e/references/bug-bash.md b/skills/e2e/references/bug-bash.md index a75eacd82d..1bc48ddd26 100644 --- a/skills/e2e/references/bug-bash.md +++ b/skills/e2e/references/bug-bash.md @@ -103,8 +103,11 @@ export default { ``` Set `BUGBASH_PASSWORD` to 6 or more characters first, or the config fails to -load. Pass `--config e2e.bugbash.config.ts` to every command below, and -`open_session {config: "e2e.bugbash.config.ts"}` over MCP. +load: a shorter password can't be redacted without rewriting ordinary text, +so an account the app seeds with one (`admin`/`admin`) needs a longer +password before it can be a credential. Pass `--config e2e.bugbash.config.ts` +to every command below, and `open_session {config: "e2e.bugbash.config.ts"}` +over MCP. ## 2. Plan charters @@ -151,9 +154,14 @@ CHARTERS while IFS='|' read -r slug target agent charter; do [ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter" done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \ - 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 6 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ + 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 15 --timeout 900000 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ ``` +`--timeout` ends a charter with its assessment whatever its step count, and +a step takes about a minute, so the flags above give each charter about +fifteen minutes. Under a time limit, size the first round to end by about a +third of it. + The log's summary prints `AI` (cost) and `Duration`. Exit code `1` means issues were reported: read the log either way. Exit codes `2` and `3` are setup and environment problems; fix them and rerun that charter alone. @@ -188,40 +196,70 @@ Sort every finding before writing any test. Read the source to sort them. A finding is a model's claim; prove each candidate before reporting it. When your client can start subagents (Claude Code's Agent or Task tool, for one), start one per area with its three to five candidates as the log printed -them and these steps, up to four at a time, the `e2e mcp` server's default -session limit. Each reports back, per finding: confirmed or rejected, the -root cause as `file:line` when it may read the source, the repro test path, -and the failure it saw. Without subagents, verify one area after another. +them and these steps, up to four at a time (the `e2e mcp` server's default +session limit, which step 4 uses). Each reports back, per finding: +confirmed or rejected, the root cause as `file:line` when it may read the +source, the repro test path, and the failure it saw. Without subagents, +verify one area after another. 1. Read `actual` against the screenshot, or the video when there is none. A finding the evidence contradicts is rejected here. Settle the artifact bucket first: a "dead" link's destination must actually open (a valid href with a prevented default is still dead), a copy claim must show in the rendered screenshot. -2. Write a repro test that follows the reproduction and asserts the - expected behavior, so it fails today and passes once the bug is fixed. - Put it under `bugbash/` inside the directory the config's `tests` glob - covers (`tests/bugbash/.e2e.ts` for the default): a file the glob - does not match is never selected, whatever path you pass to `e2e run`. - Build on the project's fixtures (a `test.extend` fixture, a setup test's - session). Prefer `screen` actions and `expect` with exact values; - `agent.act` for a step that varies, `agent.assert` for an outcome only - judgment can check (topic `writing-tests`). Tag it `{ tags: ['bugbash'] }`. -3. Get exact locators from the live app: with the `e2e mcp` server - registered, `open_session` with the bug-bash config, pass its session id - to every call, walk the reproduction, and `locate` each locator before - writing it (topic `mcp`). Close your session when done. -4. Run the file alone: `npx e2e run tests/bugbash/.e2e.ts`. Confirmed - only when it fails with `ASSERTION_FAILED` on the assertion that encodes - the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, a setup - error) means the test is wrong: fix it and rerun. A passing test means - the bug did not reproduce: reject the finding, say so, and move the test - out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as - a regression test). Only failing repro tests stay there. +2. Write a repro test from the finding: its reproduction as `agent.act` + steps and its expected behavior as the closing `agent.assert`, so it + fails today and passes once the bug is fixed. It needs no locators. Sign + in as the charter did, with its credential as step parameters or a setup + test's session. Assert only the defect: a count or a record another repro + changes makes it fail for the wrong reason. Put it under `bugbash/` + inside the directory the config's `tests` glob covers + (`tests/bugbash/.e2e.ts` for the default): a file the glob does not + match is never selected, whatever path you pass to `e2e run`. Tag it + `{ tags: ['bugbash'] }`. + + ```ts + // tests/bugbash/cart-line-quantity.e2e.ts + import { test } from '@e2e-dev/web'; + import { credentials } from 'e2e'; + + test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => { + const shopper = credentials.user('bb-cart'); + await app.open('/'); + await agent.act('sign in with the given credentials', { + params: { username: shopper.username, password: shopper.password }, + }); + await agent.assert('the app shows you are signed in'); + await agent.act('add "Dune" to the cart'); + await agent.act('set the quantity of "Dune" in the cart to 2'); + await agent.assert('the cart line for "Dune" totals twice its unit price'); + }); + ``` +3. Run your repros with an output directory of your own, `npx e2e run + tests/bugbash/.e2e.ts --output .e2e/bugbash/verify-` (a run + clears its output's results, and another subagent may be reading them), + then one file alone while you fix it. Confirmed when it fails on the + reported defect: `ASSERTION_FAILED` from the closing `assert`, or a + failed `act` whose message names the defect. A failed `act` carries the + code the model chose, so only its message and screenshot tell the defect + apart from a failure short of it. A blocked `act`, a timeout, a setup + error, or a failure that names something else means the repro is wrong: + sharpen the step and rerun. A passing test means the bug did not + reproduce: reject the finding, say so, and move the test out of + `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a + regression test). Only failing repro tests stay there. +4. When the user wants a scripted regression test, harden a confirmed repro + with `screen` actions and exact `expect` values (topic `writing-tests`): + with the `e2e mcp` server registered, `open_session` with the bug-bash + config, pass its session id to every call, walk the reproduction, and + `locate` each locator before writing it (topic `mcp`). Close your session + when done. A scripted step that opens a native dialog needs + `browser.onDialog` first. ## 7. Report -Lead with the confirmed bugs, most severe first. For each: title, path, one +Run every repro once, `npx e2e run --tag bugbash`, so each bug you report +still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one line of expected against actual, root cause when known, steps, screenshot and video paths, repro test, and the charters that found it. Then the environment candidates that are production risks, marked unverified. Then @@ -239,6 +277,10 @@ regression test with its `bugbash` tag removed. - Never report an unverified finding as a bug. "The explorer reported" is not "confirmed". +- Under a time limit, spend it: once a round is verified, run charters for + the areas and postures no charter reached, until about a tenth of the + time is left. Report each bug as soon as its repro confirms it, so a bash + cut off at the limit keeps it. - One charter, one area. A charter that spans the whole app ends at its step budget having skimmed everything. - Seeded data and test accounts the app ships for development are not From 25c9dac59951a665d523955c28ff6686275cbcaf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Oskar=20Kwas=CC=81niewski?= Date: Fri, 6 Nov 2026 22:38:21 +0100 Subject: [PATCH 2/2] docs(skills): run bug bash repros with the bug-bash config --- plugins/e2e/skills/e2e/references/bug-bash.md | 13 +++++++------ skills/e2e/references/bug-bash.md | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/plugins/e2e/skills/e2e/references/bug-bash.md b/plugins/e2e/skills/e2e/references/bug-bash.md index 1bc48ddd26..9e260aa66b 100644 --- a/plugins/e2e/skills/e2e/references/bug-bash.md +++ b/plugins/e2e/skills/e2e/references/bug-bash.md @@ -235,10 +235,11 @@ verify one area after another. await agent.assert('the cart line for "Dune" totals twice its unit price'); }); ``` -3. Run your repros with an output directory of your own, `npx e2e run - tests/bugbash/.e2e.ts --output .e2e/bugbash/verify-` (a run - clears its output's results, and another subagent may be reading them), - then one file alone while you fix it. Confirmed when it fails on the +3. Run your repros with the bug-bash config and an output directory of your + own, `npx e2e run tests/bugbash/.e2e.ts --config + e2e.bugbash.config.ts --output .e2e/bugbash/verify-` (a run clears + its output's results, and another subagent may be reading them), then one + file alone while you fix it. Confirmed when it fails on the reported defect: `ASSERTION_FAILED` from the closing `assert`, or a failed `act` whose message names the defect. A failed `act` carries the code the model chose, so only its message and screenshot tell the defect @@ -258,8 +259,8 @@ verify one area after another. ## 7. Report -Run every repro once, `npx e2e run --tag bugbash`, so each bug you report -still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one +Run every repro once, `npx e2e run --tag bugbash --config +e2e.bugbash.config.ts`, so each bug you report still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one line of expected against actual, root cause when known, steps, screenshot and video paths, repro test, and the charters that found it. Then the environment candidates that are production risks, marked unverified. Then diff --git a/skills/e2e/references/bug-bash.md b/skills/e2e/references/bug-bash.md index 1bc48ddd26..9e260aa66b 100644 --- a/skills/e2e/references/bug-bash.md +++ b/skills/e2e/references/bug-bash.md @@ -235,10 +235,11 @@ verify one area after another. await agent.assert('the cart line for "Dune" totals twice its unit price'); }); ``` -3. Run your repros with an output directory of your own, `npx e2e run - tests/bugbash/.e2e.ts --output .e2e/bugbash/verify-` (a run - clears its output's results, and another subagent may be reading them), - then one file alone while you fix it. Confirmed when it fails on the +3. Run your repros with the bug-bash config and an output directory of your + own, `npx e2e run tests/bugbash/.e2e.ts --config + e2e.bugbash.config.ts --output .e2e/bugbash/verify-` (a run clears + its output's results, and another subagent may be reading them), then one + file alone while you fix it. Confirmed when it fails on the reported defect: `ASSERTION_FAILED` from the closing `assert`, or a failed `act` whose message names the defect. A failed `act` carries the code the model chose, so only its message and screenshot tell the defect @@ -258,8 +259,8 @@ verify one area after another. ## 7. Report -Run every repro once, `npx e2e run --tag bugbash`, so each bug you report -still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one +Run every repro once, `npx e2e run --tag bugbash --config +e2e.bugbash.config.ts`, so each bug you report still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one line of expected against actual, root cause when known, steps, screenshot and video paths, repro test, and the charters that found it. Then the environment candidates that are production risks, marked unverified. Then