Skip to content

docs(skills): bug bash repros from the finding, charters sized to the clock - #1601

Merged
okwasniewski merged 2 commits into
mainfrom
oskar/bug-bash-agentic-repros
Nov 9, 2026
Merged

okwasniewski merged 2 commits into
mainfrom
oskar/bug-bash-agentic-repros

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

The bundled skill's bug bash had a coding agent write every repro test with exact locators, found one by one through e2e mcp locate, before a finding counted. That is the most expensive part of a bash and the one most likely to fail for reasons unrelated to the bug. Its charters also stopped at --max-steps 6, a number from when explore capped runs at 12 steps.

  • Repros from the finding. A repro test is the finding's reproduction as agent.act steps, one goal each, plus its expected behavior as the closing agent.assert. It needs no locators, and the charter's credential goes in as step parameters (or a setup test's session).
  • Confirmed means the repro fails on the reported defect: ASSERTION_FAILED from the closing assert, or a failed act whose message names the defect. A failed act carries the code the model chose, so the skill says to read its message and screenshot. A pass means the bug did not reproduce.
  • Assert only the defect. Repros share the app's data, so a count another repro changes fails for the wrong reason.
  • Verifiers stay apart. Each subagent runs its own area's repros with the bug-bash config into --output .e2e/bugbash/verify-<area>, since a run clears its output's results. The report runs every repro once with npx e2e run --tag bugbash --config e2e.bugbash.config.ts.
  • Scripted hardening (screen actions, exact expect, MCP locate) is a step for when the user wants a regression test. A scripted step that opens a native dialog needs browser.onDialog.
  • Charters sized to the clock: --max-steps 15 --timeout 900000, about fifteen minutes. Under a time limit the first round ends by a third of it, the bash keeps running charters for what no charter reached until a tenth is left, and each bug is reported once its repro confirms it.
  • An account the app seeds with a password under 6 characters (admin/admin) is called out before the config fails on it.
  • docs/bug-bash.mdx, docs/reference/mcp.mdx, and the create-verification-skill generator and its example now describe the same bar.
// tests/bugbash/cart-line-quantity.e2e.ts
import { test } from '@e2e-dev/web';
import { credentials } from 'e2e';

test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => {
  const shopper = credentials.user('bb-cart');
  await app.open('/');
  await agent.act('sign in with the given credentials', {
    params: { username: shopper.username, password: shopper.password },
  });
  await agent.assert('the app shows you are signed in');
  await agent.act('add "Dune" to the cart');
  await agent.act('set the quantity of "Dune" in the cart to 2');
  await agent.assert('the cart line for "Dune" totals twice its unit price');
});

Verified

Ran it locally: yes

  • The example above typechecks in the testbed (tsc --noEmit).
  • Agentic repros written this way for a scratch notes app's findings, npx e2e run --tag bugbash with openai/gpt-6-luna-fast: two real bugs failed with ASSERTION_FAILED naming the defect, one from the closing assert and one from the act that met it. The not-a-bug control passed. The three repros took 14.9 s and $0.0032 together.
  • A first draft of the control asserted a note count. It failed because another repro had deleted notes from the shared app, which is why the skill now says to assert only the defect.
  • With fix(web): answer native dialogs an agent step's action opens #1599, the same repros get past confirm() and leave-page dialogs. Without it, the confirm-gated repro ends AUTOMATION_UNSUPPORTED and the control ends ACTION_FAILED.

… clock

Repro tests start as agent.act steps plus a closing agent.assert, with no
locators; scripted hardening over MCP is opt-in. Each verifier runs its own
repros into its own output and the report runs them all once. Confirmed
means failing on the reported defect, read from the message for an act.
Charters get ~15 min (was 6 steps), a time-limited bash keeps running new
charters and reports each bug once confirmed, and short seeded passwords
are called out.
@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile label Nov 6, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@okwasniewski

Copy link
Copy Markdown
Member Author

[claude-opus-5-5] responding on behalf of okwasniewski

Left for a human:

  • Not measured: how a coding agent runs a whole bug bash with this text. Verified only the pieces (the example typechecks; agentic repros on a scratch app confirmed two real bugs and passed the control).
  • Judgment call: repros now start agentic (agent.act plus a closing agent.assert) and scripted locator hardening is opt-in, and a failed act counts as confirmed only when its message names the defect, which is a read of model text.

@okwasniewski
okwasniewski merged commit 0f158a3 into main Nov 9, 2026
30 checks passed
@okwasniewski
okwasniewski deleted the oskar/bug-bash-agentic-repros branch November 9, 2026 17:12
@github-actions github-actions Bot mentioned this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant