From 576a2bf2fc6c55cd109a780f7ad9fca59fb5165c Mon Sep 17 00:00:00 2001 From: Dario Kondratiuk Date: Tue, 10 Nov 2026 19:02:48 -0300 Subject: [PATCH 1/2] Port the bug bash repro and charter wording from upstream (#1601) Repro tests are written from the finding as ActAsync steps and a closing AssertAsync, so confirming a bug needs no locators. Verifiers confirm a bug when its repro fails on the reported defect. Charters get about fifteen minutes (--max-steps 15, --timeout 900000). Under a time limit the bash keeps running new charters and reports each bug as it is confirmed. An account with a password under 6 characters is called out before the config fails. Closes #1510 Co-Authored-By: Claude Haiku 5.5 --- docs-site/src/content/docs/docs/bug-bash.mdx | 4 +- skills/create-verification-skill/SKILL.md | 2 +- .../example/verify-todomvc/SKILL.md | 4 +- skills/e2e-dotnet/references/bug-bash.md | 94 ++++++++++++++----- 4 files changed, 75 insertions(+), 29 deletions(-) diff --git a/docs-site/src/content/docs/docs/bug-bash.mdx b/docs-site/src/content/docs/docs/bug-bash.mdx index acbac8106..d17553b2b 100644 --- a/docs-site/src/content/docs/docs/bug-bash.mdx +++ b/docs-site/src/content/docs/docs/bug-bash.mdx @@ -39,8 +39,8 @@ Bug bash the checkout and account settings on this branch. 3. Writes `e2e.bugbash.config.json`, a copy of your config with one agent for each posture (`--agent`), then runs one `e2e explore` for each charter, 4 at the same time, each with its own output directory (`--output`). 4. Merges the findings of all charters. 5. Rejects findings that have a known cause. The cause can be a limit of the explorer (for example, a link that opens a new tab), the local environment, the intended design, or the seed data. -6. Writes a repro test that asserts the expected behavior for each remaining finding. It can start one subagent per area, each with its own [`e2e mcp` session](/coding-agents/) to check locators. -7. Runs each repro test. The finding is a bug only when the test fails with `ASSERTION_FAILED`. +6. Writes a repro test for each remaining finding: the finding's steps as `Agent.ActAsync` calls and its expected behavior as the closing `Agent.AssertAsync`. It can start one subagent per area. +7. Runs the repro tests. The finding is a bug only when its test fails on the reported defect: `ASSERTION_FAILED` from the closing `Agent.AssertAsync`, or an `Agent.ActAsync` that fails on the defect itself. 8. Reports, in this order: confirmed bugs, production risks that it cannot verify locally, rejected findings by reason, and warnings. Each confirmed bug has its repro test and steps. It also has the explorer's screenshot and video when the engine records them. diff --git a/skills/create-verification-skill/SKILL.md b/skills/create-verification-skill/SKILL.md index f0c4474a2..c0ddc34ac 100644 --- a/skills/create-verification-skill/SKILL.md +++ b/skills/create-verification-skill/SKILL.md @@ -75,7 +75,7 @@ Every section says what the interview found, no placeholders. | Doctor | One read-only check that answers "is this instance worth driving?". The smoke test is a model-free test that opens the app and pins one landmark with `Expect.That`, kept out of the `RealModel` category; `dotnet test --filter "FullyQualifiedName~SmokeTests"` exits `0` when the app answers, the engine runs, and the landmark renders. No smoke test in the project: write one now. A signed-in app's auth doctor is its sign-in test. An agent runs Doctor first whenever anything looks off | | Drive | Two lanes and when to use each. Live over MCP: `open_session`, `observe`, the verbs, and `locate` before any locator is written (topic `mcp`); list the credentials the session offers through `type_secret`. Scripted: a test class in the test project, `Screen` where the name on screen is known, `Agent.ActAsync` where the flow varies (topic `writing-tests`, and `writing-tests-nunit` or `writing-tests-xunit` for the framework the project uses). Prefer stable handles, roles and accessible names, labels, test ids, routes. Name the credentials a signed-in drive starts from (`Credentials.User(name)`) and the environment variables that hold them, `E2E_USER__USERNAME` and `E2E_USER__PASSWORD`, never their values | | Evidence | What a proof keeps and where. The port writes no test report. A trace-enabled attempt writes `.e2e/results//trace.md` when its trace mode keeps it (`retain-on-failure` keeps failed attempts, `on` every attempt; topic `debugging`), and a video is written only when video recording is on. The proof is the `dotnet test` result (`--logger "trx;LogFileName=.trx"`, topic `running`) and what the test pins: `Expect.That(...)` on the end state, and `ToHaveScreenshotAsync` for a stored screenshot (topic `writing-tests`). The next run reuses `.e2e/` for the replay cache; copy a result out before a run that overwrites it. State the proof standards (below) | -| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (`e2e.bugbash.config.json`: what the local app cannot do, the postures as agents, the isolation choice from the interview), the accounts to seed with their `E2E_USER__PASSWORD` variables, the `.e2e/bugbash/` line added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once a repro test in the test project's `BugBash` folder fails with `ASSERTION_FAILED` on the assertion that encodes it | +| Bug bash | How this app is bashed (topic `bug-bash`): the bug-bash config to write (`e2e.bugbash.config.json`: what the local app cannot do, the postures as agents, the isolation choice from the interview), the accounts to seed with their `E2E_USER__PASSWORD` variables, the `.e2e/bugbash/` line added to `.gitignore`, and the rule that the charters come from the feature map (section 3). The verification bar: a finding is a bug only once a repro test in the test project's `BugBash` folder fails on the reported defect, as topic `bug-bash` step 6 reads it | | Cleanup | What to tear down. `close_session` stops a session's browser; an app the user started stays theirs. Never kill by process name; stop what you started. Cleanup removes instances and scratch state (seeded accounts, scaffolding, repro tests whose findings were all rejected), never the evidence; name where the evidence is | | Helpers | Any script the skill ships is executable and its invocation is in the skill body. A helper the reader has to reverse-engineer is not a helper. Most projects need none; `dotnet test` and the `e2e` tool are the helpers | diff --git a/skills/create-verification-skill/references/example/verify-todomvc/SKILL.md b/skills/create-verification-skill/references/example/verify-todomvc/SKILL.md index 45e194142..c3e8beb4e 100644 --- a/skills/create-verification-skill/references/example/verify-todomvc/SKILL.md +++ b/skills/create-verification-skill/references/example/verify-todomvc/SKILL.md @@ -65,8 +65,8 @@ in [features/](features/), one H2 per feature, already in the reads. The app is hosted, so several explorers need no isolation change, and the list is per browser context, so they do not collide. `.e2e/bugbash/` is in `.gitignore`. A finding is a bug once a repro test in a `BugBash` folder of -the test project fails with `ASSERTION_FAILED` on the assertion that encodes -it (topic `bug-bash`). Give each repro its own data with `Values.Unique(...)`. +the test project fails on the reported defect, as topic `bug-bash` step 6 +reads it. Give each repro its own data with `Values.Unique(...)`. ## Cleanup diff --git a/skills/e2e-dotnet/references/bug-bash.md b/skills/e2e-dotnet/references/bug-bash.md index 95bc18e9b..2cc7bad60 100644 --- a/skills/e2e-dotnet/references/bug-bash.md +++ b/skills/e2e-dotnet/references/bug-bash.md @@ -98,7 +98,10 @@ Save it as `e2e.bugbash.config.json`: The target declares no `app.command`, because the app already runs. The `credentials` hold no password: set `E2E_USER_BB_CART_PASSWORD` and `E2E_USER_BB_ACCOUNT_PASSWORD` (the seed script's password, 6 or more -characters) in the environment, so the file stays free of secrets. The three +characters) in the environment, so the file stays free of secrets. A shorter +password can't be redacted without rewriting ordinary text, so an account the +app seeds with one (`admin`/`admin`) needs a longer password before it can be +a credential. The three agents are the postures from step 2 as personas: the same model and budgets, a different stance. An agent inherits nothing from `default` (topic `agent`), so each repeats the model, budgets, and `context`. Each charter picks one with @@ -154,9 +157,14 @@ CHARTERS while IFS='|' read -r slug target agent charter; do [ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter" done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \ - 'e2e explore "$4" --config e2e.bugbash.config.json --target "$2" --agent "$3" --output ".e2e/bugbash/$1/results" --max-steps 6 --video=on < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ + 'e2e explore "$4" --config e2e.bugbash.config.json --target "$2" --agent "$3" --output ".e2e/bugbash/$1/results" --max-steps 15 --timeout 900000 --video=on < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ ``` +`--timeout` ends a charter with its assessment whatever its step count, and +a step takes about a minute, so the flags above give each charter about +fifteen minutes. Under a time limit, size the first round to end by about a +third of it. + The log's summary prints the findings and the steps. Exit code `1` means issues were reported (or nothing was explored): read the log either way. Exit codes `2` and `3` are setup and environment problems; fix them and @@ -202,28 +210,60 @@ and the failure it saw. Without subagents, verify one area after another. bucket first: a "dead" link's destination must actually open (a valid href with a prevented default is still dead), a copy claim must show in the rendered screenshot. -2. Write a repro test that follows the reproduction and asserts the - expected behavior, so it fails today and passes once the bug is fixed. - Put it in the test project, in a `BugBash` folder with one class per - area (`BugBash/CartTests.cs`), so the repro tests are easy to find and to - leave out. Build on the project's fixtures (its test base class, its - sign-in helper). Prefer `Screen` actions and `Expect.That` with exact - values; `Agent.ActAsync` for a step that varies, `Agent.AssertAsync` for - an outcome only judgment can check (topic `writing-tests`). Tag it - `bugbash`: NUnit `[Category("bugbash")]`, xUnit v3 +2. Write a repro test from the finding: its reproduction as `Agent.ActAsync` + steps and its expected behavior as the closing `Agent.AssertAsync`, so it + fails today and passes once the bug is fixed. It needs no locators. Sign + in as the charter did, with its credential as step parameters or a setup + test's session. Assert only the defect: a count or a record another repro + changes makes it fail for the wrong reason. Put it in the test project, + in a `BugBash` folder with one class per area + (`BugBash/CartTests.cs`), so the repro tests are easy to find and to + leave out. Tag it `bugbash`: NUnit `[Category("bugbash")]`, xUnit v3 `[Trait("TestCategory", "bugbash")]`. Give it its own data (`Values.Unique(...)`). -3. Get exact locators from the live app: with the `e2e mcp` server - registered, `open_session` with the bug-bash config, pass its session id - to every call, walk the reproduction, and `locate` each locator before - writing it (topic `mcp`). Close your session when done. -4. Run the class alone: `dotnet test --filter "FullyQualifiedName~BugBash.CartTests"`. - Confirmed only when it fails with `ASSERTION_FAILED` on the assertion - that encodes the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, - a setup error) means the test is wrong: fix it and rerun. A passing test - means the bug did not reproduce: reject the finding, say so, and delete - the test (or offer it as a regression test). Only failing repro tests - stay in `BugBash/`. + + ```csharp + // BugBash/CartTests.cs + public sealed class CartTests : E2ETest + { + [Test] + [Category("bugbash")] + public async Task A_cart_line_counts_its_quantity() + { + var shopper = Credentials.User("bb-cart"); + await App.OpenAsync("/"); + await Agent.ActAsync("sign in with the given credentials", new ActOptions + { + Params = new Dictionary + { + ["username"] = shopper.Username, + ["password"] = shopper.Password, + }, + }); + await Agent.AssertAsync("the app shows you are signed in"); + await Agent.ActAsync("add \"Dune\" to the cart"); + await Agent.ActAsync("set the quantity of \"Dune\" in the cart to 2"); + await Agent.AssertAsync("the cart line for \"Dune\" totals twice its unit price"); + } + } + ``` +3. Run the class alone: `dotnet test --filter "FullyQualifiedName~BugBash.CartTests"`. + Confirmed when it fails on the reported defect: `ASSERTION_FAILED` from + the closing `Agent.AssertAsync`, or a failed `Agent.ActAsync` whose + message names the defect. A failed `ActAsync` carries the code the model + chose, so only its message and screenshot tell the defect apart from a + failure short of it. A blocked act, a timeout, a setup error, or a + failure that names something else means the repro is wrong: sharpen the + step and rerun. A passing test means the bug did not reproduce: reject + the finding, say so, and delete the test (or offer it as a regression + test). Only failing repro tests stay in `BugBash/`. +4. When the user wants a scripted regression test, harden a confirmed repro + with `Screen` actions and exact `Expect.That` values (topic + `writing-tests`): with the `e2e mcp` server registered, `open_session` + with the bug-bash config, pass its session id to every call, walk the + reproduction, and `locate` each locator before writing it (topic `mcp`). + Close your session when done. A scripted step that opens a native dialog + needs `Browser.OnDialogAsync` first. ## 7. Report @@ -235,8 +275,10 @@ the rejected findings grouped by reason (environment, design, fixture, did not reproduce), and the warnings. End with the charters run, their cost, and the areas no charter reached. -The repro tests fail until the bugs are fixed: the gating run leaves them out -(`dotnet test --filter "TestCategory!=bugbash"`) or they stay uncommitted. +Run every repro once, `dotnet test --filter "TestCategory=bugbash"`, so each +bug you report still fails. The repro tests fail until the bugs are fixed: +the gating run leaves them out (`dotnet test --filter "TestCategory!=bugbash"`) +or they stay uncommitted. Offer to fix each bug: the repro test turning green is the proof, and it stays as the regression test with its `bugbash` tag removed. @@ -244,6 +286,10 @@ stays as the regression test with its `bugbash` tag removed. - Never report an unverified finding as a bug. "The explorer reported" is not "confirmed". +- Under a time limit, spend it: once a round is verified, run charters for + the areas and postures no charter reached, until about a tenth of the + time is left. Report each bug as soon as its repro confirms it, so a bash + cut off at the limit keeps it. - One charter, one area. A charter that spans the whole app ends at its step budget having skimmed everything. - Seeded data and test accounts the app ships for development are not From d092450c0a3c61617b237a9690198e04b1545eba Mon Sep 17 00:00:00 2001 From: Dario Kondratiuk Date: Tue, 10 Nov 2026 19:05:20 -0300 Subject: [PATCH 2/2] Reflow the bug bash password note into its own paragraph Co-Authored-By: Claude Haiku 5.5 --- skills/e2e-dotnet/references/bug-bash.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/skills/e2e-dotnet/references/bug-bash.md b/skills/e2e-dotnet/references/bug-bash.md index 2cc7bad60..db585e33a 100644 --- a/skills/e2e-dotnet/references/bug-bash.md +++ b/skills/e2e-dotnet/references/bug-bash.md @@ -101,12 +101,13 @@ The target declares no `app.command`, because the app already runs. The characters) in the environment, so the file stays free of secrets. A shorter password can't be redacted without rewriting ordinary text, so an account the app seeds with one (`admin`/`admin`) needs a longer password before it can be -a credential. The three -agents are the postures from step 2 as personas: the same model and budgets, -a different stance. An agent inherits nothing from `default` (topic `agent`), -so each repeats the model, budgets, and `context`. Each charter picks one with -`--agent` (step 3). The config has no `tests` or `reporters` key, so repro tests live in the test -project (step 6). Pass `--config e2e.bugbash.config.json` to every +a credential. + +The three agents are the postures from step 2 as personas: the same model and +budgets, a different stance. An agent inherits nothing from `default` (topic +`agent`), so each repeats the model, budgets, and `context`. Each charter picks +one with `--agent` (step 3). The config has no `tests` or `reporters` key, so +repro tests live in the test project (step 6). Pass `--config e2e.bugbash.config.json` to every command below, and `open_session {config: "e2e.bugbash.config.json"}` over MCP.