Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/skill-bug-bash-repros.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

The bundled e2e skill's bug bash writes each repro test from the finding as `agent.act` steps and a closing `agent.assert`, so confirming a bug needs no locators. Each verifier runs its own repros into its own output directory, and the report runs them all once with `--tag bugbash`. A finding is confirmed when its repro fails on the reported defect, and the skill says how to read a failed `act`. Charters get about fifteen minutes each, a bash under a time limit keeps running new charters until near the end and reports each bug as its repro confirms it, and an account with a password under 6 characters is called out before the config fails.
11 changes: 6 additions & 5 deletions docs/bug-bash.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,12 @@ Bug bash the checkout and account settings on this branch.
5. Rejects findings that have a known cause. The cause can be a limit of
the explorer (for example, a link that opens a new tab), the local
environment, the intended design, or the seed data.
6. Writes a repro test that asserts the expected behavior for each
remaining finding. It can start one subagent per area, each with its own
[`e2e mcp` session](/mcp-server) to check locators.
7. Runs each repro test. The finding is a bug only when the test fails with
`ASSERTION_FAILED`.
6. Writes a repro test for each remaining finding: the finding's steps as
`agent.act` calls and its expected behavior as the closing
`agent.assert`. It can start one subagent per area.
7. Runs the repro tests. The finding is a bug only when its test fails on
the reported defect: `ASSERTION_FAILED` from the closing check, or an
`act` that fails on the defect itself.
8. Reports, in this order: confirmed bugs, production risks that it cannot
verify locally, rejected findings by reason, and warnings.

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/mcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -652,8 +652,8 @@ Several sessions can be open at once, up to `--max-sessions`, each on its
own attempt and its own browser or device. Parallel agents, such as a
coding agent's subagents, share one server this way: each opens a session
and passes its id to every `tools`, `call`, and `close_session`.
[Bug bashes](/bug-bash) use this: every verifying subagent drives its own
session.
[Bug bashes](/bug-bash) use this: a verifying subagent that hardens a repro
into a scripted test drives its own session.

| Throws | When |
| --- | --- |
Expand Down
2 changes: 1 addition & 1 deletion plugins/e2e/skills/create-verification-skill/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ found, no placeholders.
| Doctor | One read-only check that answers "is this instance worth driving?". `npx e2e list` loads the config and prints every test-target pair without starting anything, so a config error shows here first. Then the smoke test, a model-free test that opens the app and pins one landmark; exit `0` means the app starts, the engine runs, and the landmark renders. No smoke test in the project: write `tests/smoke.e2e.ts` now. A signed-in app's auth doctor is an ordinary signed-in smoke test that consumes a named session: selecting that test runs its setup and checks the restored state, so one command proves the app starts, the session restores, and a signed-in landmark renders. Running the setup file alone selects no test and exits `NO_TESTS`, so it is not the doctor. An agent runs Doctor first whenever anything looks off |
| Drive | Two lanes and when to use each. Live over MCP: `open_session`, `observe`, the verbs, and `locate` before any locator is written (topic `mcp`); list the project tools and credentials the session's catalog offers. Scripted: a test under the `tests` glob, `screen` where the name on screen is known, `agent.act` where the flow varies (topic `writing-tests`). Prefer stable handles, roles and accessible names, labels, test ids, routes. Name the credentials and sessions a signed-in drive starts from |
| Evidence | What a proof keeps and where. Every run writes `.e2e/report.json`; a failed test gets `.e2e/results/<test>/trace.md`, `--trace on` gives one to passing tests too, `--video` records the attempt (topic `running`), `app.screenshot('<name>')` saves a named frame (topic `writing-tests`). The next run clears `.e2e/results/`, Doctor included, so a proof that must outlive it runs with `--output .e2e/proof/<name>` or its attempt directory is copied out before anything else runs. State the proof standards (below) |
| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/<slug>.e2e.ts` fails with `ASSERTION_FAILED` on the assertion that encodes it |
| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/<slug>.e2e.ts` fails on the reported defect, as topic `bug-bash` step 6 reads it |
| Cleanup | What to tear down. The runner stops the app it started and `close_session` stops the one a session started; a server the user started stays theirs. Never kill by process name; stop what you started. Cleanup removes instances and scratch state (seeded accounts, scaffolding, charters whose findings were all rejected), never the evidence; name where the evidence is |
| Helpers | Any script the skill ships is executable and its invocation is in the skill body. A helper the reader has to reverse-engineer is not a helper. Most projects need none; the e2e CLI is the helper |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ fan-out in topic `bug-bash` step 3 reads. `.e2e/bugbash/` is in
`.gitignore`, with `.e2e/proof/`. One explorer at a time needs no isolation change; several
need `pnpm app` started first and `reuseExisting: true` on the command,
since the port is fixed. A finding is a bug once
`tests/bugbash/<slug>.e2e.ts` fails with `ASSERTION_FAILED` on the
assertion that encodes it; anything else is the explorer's claim.
`tests/bugbash/<slug>.e2e.ts` fails on the reported defect, as topic
`bug-bash` step 6 reads it; anything else is the explorer's claim.

## Cleanup

Expand Down
99 changes: 71 additions & 28 deletions plugins/e2e/skills/e2e/references/bug-bash.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,11 @@ export default {
```

Set `BUGBASH_PASSWORD` to 6 or more characters first, or the config fails to
load. Pass `--config e2e.bugbash.config.ts` to every command below, and
`open_session {config: "e2e.bugbash.config.ts"}` over MCP.
load: a shorter password can't be redacted without rewriting ordinary text,
so an account the app seeds with one (`admin`/`admin`) needs a longer
password before it can be a credential. Pass `--config e2e.bugbash.config.ts`
to every command below, and `open_session {config: "e2e.bugbash.config.ts"}`
over MCP.

## 2. Plan charters

Expand Down Expand Up @@ -151,9 +154,14 @@ CHARTERS
while IFS='|' read -r slug target agent charter; do
[ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter"
done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \
'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 6 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _
'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 15 --timeout 900000 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _
```

`--timeout` ends a charter with its assessment whatever its step count, and
a step takes about a minute, so the flags above give each charter about
fifteen minutes. Under a time limit, size the first round to end by about a
third of it.

The log's summary prints `AI` (cost) and `Duration`. Exit code `1` means
issues were reported: read the log either way. Exit codes `2` and `3` are
setup and environment problems; fix them and rerun that charter alone.
Expand Down Expand Up @@ -188,40 +196,71 @@ Sort every finding before writing any test. Read the source to sort them.
A finding is a model's claim; prove each candidate before reporting it. When
your client can start subagents (Claude Code's Agent or Task tool, for one),
start one per area with its three to five candidates as the log printed
them and these steps, up to four at a time, the `e2e mcp` server's default
session limit. Each reports back, per finding: confirmed or rejected, the
root cause as `file:line` when it may read the source, the repro test path,
and the failure it saw. Without subagents, verify one area after another.
them and these steps, up to four at a time (the `e2e mcp` server's default
session limit, which step 4 uses). Each reports back, per finding:
confirmed or rejected, the root cause as `file:line` when it may read the
source, the repro test path, and the failure it saw. Without subagents,
verify one area after another.

1. Read `actual` against the screenshot, or the video when there is none.
A finding the evidence contradicts is rejected here. Settle the artifact
bucket first: a "dead" link's destination must actually open (a valid
href with a prevented default is still dead), a copy claim must show in
the rendered screenshot.
2. Write a repro test that follows the reproduction and asserts the
expected behavior, so it fails today and passes once the bug is fixed.
Put it under `bugbash/` inside the directory the config's `tests` glob
covers (`tests/bugbash/<slug>.e2e.ts` for the default): a file the glob
does not match is never selected, whatever path you pass to `e2e run`.
Build on the project's fixtures (a `test.extend` fixture, a setup test's
session). Prefer `screen` actions and `expect` with exact values;
`agent.act` for a step that varies, `agent.assert` for an outcome only
judgment can check (topic `writing-tests`). Tag it `{ tags: ['bugbash'] }`.
3. Get exact locators from the live app: with the `e2e mcp` server
registered, `open_session` with the bug-bash config, pass its session id
to every call, walk the reproduction, and `locate` each locator before
writing it (topic `mcp`). Close your session when done.
4. Run the file alone: `npx e2e run tests/bugbash/<slug>.e2e.ts`. Confirmed
only when it fails with `ASSERTION_FAILED` on the assertion that encodes
the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, a setup
error) means the test is wrong: fix it and rerun. A passing test means
the bug did not reproduce: reject the finding, say so, and move the test
out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as
a regression test). Only failing repro tests stay there.
2. Write a repro test from the finding: its reproduction as `agent.act`
steps and its expected behavior as the closing `agent.assert`, so it
fails today and passes once the bug is fixed. It needs no locators. Sign
in as the charter did, with its credential as step parameters or a setup
test's session. Assert only the defect: a count or a record another repro
changes makes it fail for the wrong reason. Put it under `bugbash/`
inside the directory the config's `tests` glob covers
(`tests/bugbash/<slug>.e2e.ts` for the default): a file the glob does not
match is never selected, whatever path you pass to `e2e run`. Tag it
`{ tags: ['bugbash'] }`.

```ts
// tests/bugbash/cart-line-quantity.e2e.ts
import { test } from '@e2e-dev/web';
import { credentials } from 'e2e';

test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => {
const shopper = credentials.user('bb-cart');
await app.open('/');
await agent.act('sign in with the given credentials', {
params: { username: shopper.username, password: shopper.password },
});
await agent.assert('the app shows you are signed in');
await agent.act('add "Dune" to the cart');
await agent.act('set the quantity of "Dune" in the cart to 2');
await agent.assert('the cart line for "Dune" totals twice its unit price');
});
```
3. Run your repros with the bug-bash config and an output directory of your
own, `npx e2e run tests/bugbash/<slug>.e2e.ts --config
e2e.bugbash.config.ts --output .e2e/bugbash/verify-<area>` (a run clears
its output's results, and another subagent may be reading them), then one
file alone while you fix it. Confirmed when it fails on the
reported defect: `ASSERTION_FAILED` from the closing `assert`, or a
failed `act` whose message names the defect. A failed `act` carries the
code the model chose, so only its message and screenshot tell the defect
apart from a failure short of it. A blocked `act`, a timeout, a setup
error, or a failure that names something else means the repro is wrong:
sharpen the step and rerun. A passing test means the bug did not
reproduce: reject the finding, say so, and move the test out of
`tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a
regression test). Only failing repro tests stay there.
4. When the user wants a scripted regression test, harden a confirmed repro
with `screen` actions and exact `expect` values (topic `writing-tests`):
with the `e2e mcp` server registered, `open_session` with the bug-bash
config, pass its session id to every call, walk the reproduction, and
`locate` each locator before writing it (topic `mcp`). Close your session
when done. A scripted step that opens a native dialog needs
`browser.onDialog` first.

## 7. Report

Lead with the confirmed bugs, most severe first. For each: title, path, one
Run every repro once, `npx e2e run --tag bugbash --config
e2e.bugbash.config.ts`, so each bug you report still fails. Lead with the confirmed bugs, most severe first. For each: title, path, one
line of expected against actual, root cause when known, steps, screenshot
and video paths, repro test, and the charters that found it. Then the
environment candidates that are production risks, marked unverified. Then
Expand All @@ -239,6 +278,10 @@ regression test with its `bugbash` tag removed.

- Never report an unverified finding as a bug. "The explorer reported" is
not "confirmed".
- Under a time limit, spend it: once a round is verified, run charters for
the areas and postures no charter reached, until about a tenth of the
time is left. Report each bug as soon as its repro confirms it, so a bash
cut off at the limit keeps it.
- One charter, one area. A charter that spans the whole app ends at its
step budget having skimmed everything.
- Seeded data and test accounts the app ships for development are not
Expand Down
2 changes: 1 addition & 1 deletion skills/create-verification-skill/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ found, no placeholders.
| Doctor | One read-only check that answers "is this instance worth driving?". `npx e2e list` loads the config and prints every test-target pair without starting anything, so a config error shows here first. Then the smoke test, a model-free test that opens the app and pins one landmark; exit `0` means the app starts, the engine runs, and the landmark renders. No smoke test in the project: write `tests/smoke.e2e.ts` now. A signed-in app's auth doctor is an ordinary signed-in smoke test that consumes a named session: selecting that test runs its setup and checks the restored state, so one command proves the app starts, the session restores, and a signed-in landmark renders. Running the setup file alone selects no test and exits `NO_TESTS`, so it is not the doctor. An agent runs Doctor first whenever anything looks off |
| Drive | Two lanes and when to use each. Live over MCP: `open_session`, `observe`, the verbs, and `locate` before any locator is written (topic `mcp`); list the project tools and credentials the session's catalog offers. Scripted: a test under the `tests` glob, `screen` where the name on screen is known, `agent.act` where the flow varies (topic `writing-tests`). Prefer stable handles, roles and accessible names, labels, test ids, routes. Name the credentials and sessions a signed-in drive starts from |
| Evidence | What a proof keeps and where. Every run writes `.e2e/report.json`; a failed test gets `.e2e/results/<test>/trace.md`, `--trace on` gives one to passing tests too, `--video` records the attempt (topic `running`), `app.screenshot('<name>')` saves a named frame (topic `writing-tests`). The next run clears `.e2e/results/`, Doctor included, so a proof that must outlive it runs with `--output .e2e/proof/<name>` or its attempt directory is copied out before anything else runs. State the proof standards (below) |
| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/<slug>.e2e.ts` fails with `ASSERTION_FAILED` on the assertion that encodes it |
| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (what the local app cannot do, the postures, the isolation choice from the interview), the accounts to seed, `.e2e/bugbash/` added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once `tests/bugbash/<slug>.e2e.ts` fails on the reported defect, as topic `bug-bash` step 6 reads it |
| Cleanup | What to tear down. The runner stops the app it started and `close_session` stops the one a session started; a server the user started stays theirs. Never kill by process name; stop what you started. Cleanup removes instances and scratch state (seeded accounts, scaffolding, charters whose findings were all rejected), never the evidence; name where the evidence is |
| Helpers | Any script the skill ships is executable and its invocation is in the skill body. A helper the reader has to reverse-engineer is not a helper. Most projects need none; the e2e CLI is the helper |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ fan-out in topic `bug-bash` step 3 reads. `.e2e/bugbash/` is in
`.gitignore`, with `.e2e/proof/`. One explorer at a time needs no isolation change; several
need `pnpm app` started first and `reuseExisting: true` on the command,
since the port is fixed. A finding is a bug once
`tests/bugbash/<slug>.e2e.ts` fails with `ASSERTION_FAILED` on the
assertion that encodes it; anything else is the explorer's claim.
`tests/bugbash/<slug>.e2e.ts` fails on the reported defect, as topic
`bug-bash` step 6 reads it; anything else is the explorer's claim.

## Cleanup

Expand Down
Loading
Loading