Skip to content

fix(web): answer native dialogs an agent step's action opens - #1599

Closed
okwasniewski wants to merge 6 commits into
mainfrom
oskar/agent-native-dialogs
Closed

okwasniewski wants to merge 6 commits into
mainfrom
oskar/agent-native-dialogs

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

An agent never sees a native alert, confirm, prompt, or leave-page dialog: they sit outside the page. With no browser.onDialog handler, the web engine dismissed each one and failed the next operation with INVALID_STATE. So an agent step could not leave a form with unsaved changes or confirm a delete, live or from the replay cache.

  • DialogRouter asks its surface whether an agent step made the newest action (packages/web/src/dialogs.ts). If so, a dialog no handler takes is answered: alerts, confirms, and leave-page dialogs are accepted, and a prompt is dismissed, since the agent never sees its text. The app log gets one system line, such as confirm dialog "Delete this label?" accepted for the agent.
  • Ownership moves only on actions. PlaywrightSurface.guardAction() (packages/web/src/surface.ts) wraps every operation that acts: navigation, input, browser.evaluate, keyboard, mouse, browser.setViewport, browser.emulateMedia. It claims the dialog once the operation runs, so one refused by a pending error claims nothing. Reads (observe, locate, screenshots) and a route handler's route.fetch never take a dialog. So a test's own late dialog still fails its next step while an agent judgment polls, and a test route handler fetching mid-action can't steal the agent's dialog.
  • OperationContext.agentStep (new, optional) marks an agent step's operations, replays included (packages/e2e/src/agent/phases.ts). Replayed actions keep origin: 'test' for settling, so without it a recorded dialog step failed under --strict-cache. It crosses the engine protocol. An older core omits it and the engine falls back to origin.
  • A dialog a test's own step opens behaves as before: dismissed, and the next step fails with INVALID_STATE. A registered handler always wins.
  • docs/reference/web.mdx said an unhandled dialog throws APP_NOT_OPEN. The code throws INVALID_STATE, so the table now says that.
  • The web benchmark gets an agent scenario on /e/deferred-dialog, whose confirm opens 900 ms after the click, with its committed recording.

Why

In 201 e2e explore runs on 0.22.0 against a self-hosted Gitea (their report.json and AI traces):

  • 226 agent observations and actions failed with unhandled beforeunload dialog.
  • 114 navigate calls failed with net::ERR_ABORTED. 113 of them were the dismissed leave-page dialog.
  • 59 steps ended blocked. The model saw only Action failed: unhandled beforeunload dialog: and retried back until the step timed out.

For review

  • Confirms are accepted. The agent's action asked for the destructive step and the agent can't see the dialog to decline it. That also removes the app's last guard if an explorer clicks "Delete workspace" against its instructions. The alternative is to dismiss confirms, tell the model the text, and give it an explicit way to accept, which needs a model-visible notice channel first.
  • The model and MCP callers don't see the app log line. An MCP action's test line registers no handler, so the docs and the MCP skill now say a scripted test of that flow needs browser.onDialog. run_code is test code, so its dialogs still fail the next call. Surfacing answered dialogs in action results is a follow-up.
  • Porting to feat(web)!: WebDriver BiDi driver replaces Playwright #1503: a trial cherry-pick onto its head conflicts only in driver/dialogs.ts imports. BidiSession builds the router there and would pass the same two callbacks, and its action entry points would take guardAction's ownership line.

Verified

Ran it locally: yes

  • packages/web/tests/integration/agent-dialogs.test.ts, real Chromium, 8 cases: an agent leaves a dirty form; agent confirm, alert, and prompt; a replayed agent action (origin: 'test', agentStep: true); a test route handler's route.fetch before the agent's alert; a test's setViewport whose resize handler confirms after an agent tap, which must fail the test's next step; a test action refused by a pending route error, which must leave the agent's alert with the agent; a test's 300 ms delayed confirm while agent reads poll, which must still fail with INVALID_STATE; and a test's own confirm. On main, both live agent cases fail with unhandled ... dialog. The viewport and refused-action cases fail without their fixes.
  • Web benchmark, tests-agent deferred-dialog: recorded live once, then replayed twice with CI=1 --strict-cache: Cache 1 replayed, no model call, and the app log shows confirm dialog "Really submit the report?" accepted for the agent.
  • The whole web benchmark agent suite under --strict-cache, with the same counts on main for the existing recordings:
main this branch
Tests 33 passed 34 passed (+ deferred-dialog)
Cache 26 replayed · 2 handed off · 4 missed 27 replayed · 2 handed off · 4 missed
  • Testbed tests/dialog.e2e.ts and tests/dialogs-frames.e2e.ts (15 tests) and web benchmark tests/native-dialogs.e2e.ts and tests/deferred-dialog.e2e.ts (4 tests) pass, so scripted handlers behave as before.
  • e2e explore (openai/gpt-6-luna-fast, --max-steps 4) on a scratch notes app with a leave-page guard on its editor and a confirm() before each delete. Its delete removes the wrong note, a planted bug:
main this branch
Steps 3 of 4, all blocked, ended "steps kept failing" 4 of 4, the step limit was reached
Failed agent actions 7 (unhandled beforeunload dialog) 0
Planted delete bug not reached found: "Deleting Trip plan removes Groceries instead"
Cost $0.0107 $0.0106
  • Agentic repro tests against the same app (e2e run --tag bugbash):
Repro main this branch
Delete removes the chosen note (real bug) AUTOMATION_UNSUPPORTED: blocked by the confirm ASSERTION_FAILED: confirmed
Leaving the editor unsaved keeps the list (not a bug) ACTION_FAILED: blocked by the leave-page dialog passes

An unhandled alert, confirm, or leave-page dialog after an agent step's
action is accepted, a prompt dismissed, and the app log names it. Before,
the engine dismissed it and failed the agent's next action, so leaving a
dirty form or confirming a delete was out of reach for agent.act, explore,
and MCP action tools.

A dialog belongs to the newest action before it, never to a read, so a
test's own dialog still fails its next step even while an agent judgment
polls. OperationContext.agentStep marks agent step operations, replays
included, so a recorded dialog step replays under --strict-cache; the web
benchmark gains a deferred-dialog agent scenario with its recording.
@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 898ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 628ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 270ms
🟢 tests/async-states.e2e.ts · 2 passed 7.0s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 274ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 149ms
🟢 canvas only › a shape out of order paints the error and resets progress 125ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 448ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 274ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 174ms
🟢 tests/cart-totals.e2e.ts · 2 passed 336ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 187ms
🟢 cart totals › a quantity never drops below one 149ms
🟢 tests/checkout-review.e2e.ts · 1 passed 164ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 164ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 539ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 283ms
🟢 context menu trap › the decoy menu items refuse and keep the file 256ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.6s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 280ms
🟢 control inventory › check picks a radio the pick replaces with its summary 191ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.0s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 247ms
🟢 control inventory › setInputFiles attaches files named from the project root 141ms
🟢 control inventory › the details view is left through the browser history 198ms
🟢 control inventory › scrollIntoView brings the footnote into view 276ms
🟢 control inventory › every exercise done shows the success message 1.3s
🟢 tests/css-content-ui.e2e.ts · 2 passed 313ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 176ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 137ms
🟢 tests/date-picker.e2e.ts · 2 passed 631ms
🟢 date picker › navigates three months ahead and books the required date 355ms
🟢 date picker › confirming without the exact date shows the error 276ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.9s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 620ms
🟢 div soup › a wrong code is rejected 271ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 349ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 602ms
🟢 drag and drop › an out-of-order item resets the dropzone 297ms
🟢 drag and drop › dropping Banana then Cherry passes 305ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 222ms
🟢 filter deep link › filtering writes the query to the URL 127ms
🟢 filter deep link › opening a shared link restores the filter 95ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 507ms
🟢 gift card purchase › refuses to buy without an amount 140ms
🟢 gift card purchase › refuses a recipient without an at sign 178ms
🟢 gift card purchase › sends the gift card 189ms
🟢 tests/hover-menu.e2e.ts · 2 passed 412ms
🟢 hover menu › a decoy leaf is refused 180ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 232ms
🟢 tests/iframe-form.e2e.ts · 2 passed 529ms
🟢 iframe form › a wrong code is refused by the host page 247ms
🟢 iframe form › the code read from one frame applies through the other 282ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 468ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 100ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 170ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 198ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.7s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.7s
🟢 tests/kanban-board.e2e.ts · 3 passed 788ms
🟢 kanban board › pointer drags move both cards into the goal layout 377ms
🟢 kanban board › a swipe along a path drags a card as well 192ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 219ms
🟢 tests/login-form.e2e.ts · 4 passed 883ms
🟢 login form › rejects a malformed email 221ms
🟢 login form › rejects a short password 207ms
🟢 login form › rejects an unknown account 204ms
🟢 login form › signs in and logs out 251ms
🟢 tests/lying-labels.e2e.ts · 4 passed 747ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 152ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 126ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 230ms
🟢 lying labels › visible text drives the payment through 239ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 423ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 159ms
🟢 native dialogs › dismissing the confirm cancels the order 132ms
🟢 native dialogs › a wrong gift code is rejected 132ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.6s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 291ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 331ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.4s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.4s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 179ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 487ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 145ms
🟢 otp auto-advance › a wrong code clears every box and starts over 799ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 728ms
🟢 continues only after the banner and the overlay are gone 728ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 549ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 268ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 281ms
🟢 tests/price-sorting.e2e.ts · 3 passed 431ms
🟢 price sorting › opens in the featured order with sale prices struck through 115ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 169ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 147ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 510ms
🟢 rich text editor › bolding exactly the required word passes the check 260ms
🟢 rich text editor › bold that spills into the previous word fails the check 250ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 564ms
🟢 saved addresses › adds an address and clears the form 188ms
🟢 saved addresses › ignores an add with an empty field 159ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 217ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 566ms
🟢 shadow DOM form › rejects a wrong access code 283ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 283ms
🟢 tests/smoke.e2e.ts · 1 passed 428ms
🟢 home lists the scenarios and opens one 428ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 189ms
🟢 accepts the terms once the button clears the sticky footer 189ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 16.0s
🟢 scrolls the windowed table to the Golden Row and claims it 16.0s
🟢 tests/wishlist.e2e.ts · 2 passed 1.6s
🟢 wishlist › saving toggles the button and the tab badge 280ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.23.0 · 1m 4s · web · run artifacts

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 23 model calls · 102.4k tokens (94% cached) · $0.0041

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.7s
🟢 control inventory › check sets the checkbox and the radio 1 step 840ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 353ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 945ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.5s
🟢 control inventory › upload attaches the named project files 1 step 2.4s
🟢 control inventory › back returns from the details view to the inventory 1 step 260ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 361ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 383ms
🟢 act signs in with the benchmark account 1 step 383ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 9.4s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 4.9s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 978ms
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 3.5s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 15 calls 1m 6s
🟢 act completes shadow-dom-form 1 step · 3 calls 3.6s
🟢 act completes lying-labels 1 step · 3 calls 4.5s
🟢 act completes iframe-form 1 step · 3 calls 3.5s
🟢 act completes infinite-scroll 1 step 3.1s
🟢 act completes virtualized-table 1 step 16.1s
🟢 act completes overlay-trap 1 step 871ms
🟢 act completes sticky-chrome 1 step 905ms
🟢 act completes async-states 1 step · 3 calls 5.9s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 506ms
🟢 act completes drag-and-drop 1 step 472ms
🟢 act completes kanban-board 1 step 663ms
🟢 act completes context-menu-trap 1 step 2.7s
🟢 act completes stale-dom 1 step 527ms
🟢 act completes otp-auto-advance 1 step 761ms
🟢 act completes date-picker 1 step 892ms
🟢 act completes rich-text-editor 1 step 10.3s
🟢 act completes playbook-cleanup 1 step 656ms
🟢 act completes promo-storefront 1 step 1.4s
🟢 act completes gift-card-purchase 1 step 2.5s
🟢 act completes deferred-dialog 1 step · 3 calls 3.8s
🟢 act filters the list and the URL carries the filter 1 step 233ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 318ms
🟢 act adds an address with a run-unique label 1 step 318ms

Tested by e2e 0.23.0 · 1m 24s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/reference/web.mdx Resolved
Comment thread docs/reference/mcp.mdx Resolved
Comment thread packages/web/src/surface.ts Resolved
Comment thread skills/e2e/references/writing-tests.md Resolved
Comment thread packages/web/src/dialogs.ts Resolved
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 40 model calls · 221.0k tokens (84% cached) · $0.0137

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 40 calls 5m 43s
🟢 act completes Login Form 1 step 23.7s
🟢 act completes Infinite Scroll List 1 step 21.7s
🟢 act completes Modal Flow 1 step 10.1s
🟢 act completes Bottom Tabs 1 step 6.6s
🟢 act completes Text Input Variations 1 step 6.1s
🟢 act completes Broken Accessibility 2 steps · 5 calls 13.5s
🟢 act completes Vision Only 2 steps · 1 call 6.3s
🟢 act completes Gestures 1 step · 9 calls 22.3s
🟢 act completes Async States 1 step · 13 calls 26.2s
🟢 act completes Debounced Search 1 step 6.9s
🟢 act completes Huge Virtualized List 1 step 1m 55s
🟢 act completes Flattened Registration Form 2 steps · 1 call 7.7s
🟢 act completes Flattened Login 2 steps · 6 calls 14.1s
🟢 act completes Sticky Chrome Target 1 step 8.8s
🟢 act completes WebView Accessibility 1 step · 5 calls 14.0s
🟢 act completes Permission Prompt 1 step 5.7s
🟢 act completes Product Catalog 1 step 7.9s
🟢 act completes Error Recovery 1 step 8.7s
🟢 act completes Choice Controls 1 step 8.2s
🟢 act completes Sequential Onboarding 1 step 9.5s

Tested by e2e 0.23.0 · 5m 44s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 2s
🟢 control inventory › a label query resolves a labeled text field 8.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.7s
🟢 control inventory › a placeholder query and attribute reads find the field 4.8s
🟢 control inventory › a secure field takes a fill and denies value reads 9.2s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.9s
🟢 control inventory › toBeFocused follows the focus 10.1s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.8s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 13.3s
🟢 control inventory › a radio group reports the checked option 9.4s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.0s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.3s
🟢 control inventory › a header answers its accessible name 4.1s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.0s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.5s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.0s
🟢 control inventory › pointer verbs land at the point asked for 10.4s
🟢 control inventory › an ambiguous locator fails at once 6.5s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 12.7s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 346ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 44s
🟢 device fixture › setAppearance flips the color scheme the app reads 11.7s
🟢 device fixture › setOrientation rotates the window the app measures 7.7s
🟢 device fixture › the clipboard reads back what was written 6.3s
🟢 device fixture › home leaves the app, openApp brings it back where it was 10.4s
🟢 device fixture › closeApp then openApp relaunches on the home list 13.2s
🟢 device fixture › closeApp leaves the next commands on the same device 15.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.7s
🟢 device fixture › device.back pops the scenario 5.8s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 5.1s
🟢 device fixture › setLocation and clearLocation change what the app reads 11.3s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 11.4s
🟢 tests/device.e2e.ts · 3 passed 1m 8s
🟢 sequential onboarding echoes the exact values 40.9s
🟢 onboarding rejects a malformed email before advancing 13.8s
🟢 dismissing the keyboard over choice controls leaves them as they were 13.4s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 22s
🟢 infinite scroll reaches the target item 36.3s
🟢 product catalog adds exactly two of the right variant 10.4s
🟢 gestures: long-press, then swipe left 9.1s
🟢 sticky chrome: accept below the fold, then continue 12.8s
🟢 async states: load, pull to refresh, claim through the toast 10.8s
🟢 huge virtualized list reaches row 512 2m 49s
🟢 web view coupon form applies the code shown on the page 13.4s
🟢 tests/flows.e2e.ts · 7 passed 1m 11s
🟢 text inputs unlock the submit button 13.9s
🟢 filled text inputs report their values 11.4s
🟢 modal flow confirms through the native alert 9.8s
🟢 bottom tabs act inside the last tab 7.4s
🟢 error recovery retries, then confirms the delete 9.4s
🟢 choice controls place the exact order 7.5s
🟢 debounced search selects the target once results arrive 12.0s
🟢 tests/login-form.e2e.ts · 3 passed 49.4s
🟢 login form › rejects a malformed email 14.3s
🟢 login form › rejects an unknown account 13.9s
🟢 login form › signs in and logs out 21.2s
🟢 tests/smoke.e2e.ts · 1 passed 8.5s
🟢 home lists the scenarios and opens one 8.5s

Tested by e2e 0.23.0 · 7m 14s · ios-simulator · run artifacts

…and media changes

An action guard refuses before it runs no longer claims the next dialog,
and browser.setViewport and browser.emulateMedia claim it like other page
changes. Docs: an unhandled dialog fails the attempt at its end when no
step follows, a run_code dialog fails the next statement first, and the
skill warns that agent steps accept confirms unseen without a handler.
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 12 replayed from cache · 58 model calls · 397.8k tokens (83% cached) · $0.0262

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:266

ACTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 29.8s and 7 model calls

agent.act failed: The first long-press had already been completed. I attempted two leftward swipes across the visible “Swipe me left” card, but the screen remained on Step 2 of 3 with the card unchanged, so the swipe did not complete and I could not proceed to the final double-tap.

  • Turn 4: swipe({"from":{"x":235,"y":465},"to":{"x":55,"y":465}}) → [swipe] Swiped from (235, 465) to (55, 465).
  • Turn 5: screenshot({}) → [screenshot] The screen changed substantially since revision b8. Current screen (revision b10, path dev.e2e.benchmark / Gestures, 12 nodes):
  • Turn 6: drag_at({"x":235,"y":465,"to_x":55,"to_y":465}) → [drag_at] drag_at (235, 465) to (55, 465) failed: UNSUPPORTED_CAPABILITY: the drag from (268, 530) to (63, 530) does not join two different controls the screen…
  • Screen: dev.e2e.benchmark / Gestures

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 58 calls 8m 45s
🟢 act completes Login Form 1 step 45.0s
🟢 act completes Infinite Scroll List 1 step 40.8s
🟢 act completes Modal Flow 1 step 10.9s
🟢 act completes Bottom Tabs 1 step 7.6s
🟢 act completes Text Input Variations 1 step 13.6s
🟢 act completes Broken Accessibility 2 steps · 5 calls 25.3s
🟢 act completes Vision Only 2 steps · 6 calls 29.4s
⚠️ act completes Gestures (1 failed attempt first) 1 step · 9 calls 38.8s
🟢 act completes Async States 1 step · 10 calls 35.0s
🟢 act completes Debounced Search 1 step · 1 call 28.0s
🟢 act completes Flattened Registration Form 2 steps · 1 call 29.4s
🟢 act completes Flattened Login 2 steps · 10 calls 52.6s
🟢 act completes Sticky Chrome Target 1 step 14.0s
🟢 act completes WebView Accessibility 1 step 12.9s
🟢 act completes Permission Prompt 1 step 12.0s
🟢 act completes Photo Picker 1 step · 2 calls 30.6s
🟢 act completes Product Catalog 1 step 10.4s
🟢 act completes Error Recovery 1 step 10.0s
🟢 act completes Choice Controls 1 step · 6 calls 36.5s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 31.7s
🟢 act completes Apple Pay 1 step 10.8s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.0 · 9m 21s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 9 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 4s
🟢 control inventory › a label query resolves a labeled text field 27.8s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.0s
🟢 control inventory › a placeholder query and attribute reads find the field 6.5s
🟢 control inventory › a secure field takes a fill and denies value reads 8.1s
🟢 control inventory › press sends a character and Enter through the soft keyboard 8.6s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 9.1s
🟢 control inventory › toBeFocused follows the focus 8.4s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.2s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.1s
🟢 control inventory › a tab role answers getByRole 3.6s
🟢 control inventory › a radio group reports the checked option 8.7s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.4s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.0s
🟢 control inventory › a header answers its accessible name 4.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.9s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.9s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.5s
🟢 control inventory › a double tap lands inside the 300 ms window 6.8s
🟢 control inventory › an ambiguous locator fails at once 5.1s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.6s
🟢 refusals › verbs a device never declares are refused before the engine 4ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 19s
🟢 device fixture › setOrientation rotates the window the app measures 5.3s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.3s
🟢 device fixture › foregroundApp sees the home screen after home() 4.8s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.8s
🟢 device fixture › closeApp leaves the next commands on the same device 7.4s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.9s
🟢 device fixture › device.back pops the scenario 5.0s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.0s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.0s
🟢 device fixture › setLocation and clearLocation change what the app reads 22.0s
🟢 tests/device.e2e.ts · 5 passed 46.2s
🟢 permission prompt › allowing the system dialog enables the microphone 5.6s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 13.5s
🟢 sequential onboarding echoes the exact values 11.2s
🟢 onboarding rejects a malformed email before advancing 6.8s
🟢 dismissing the keyboard over choice controls leaves them as they were 9.2s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 31s
🟢 infinite scroll reaches the target item 35.8s
🟢 product catalog adds exactly two of the right variant 11.3s
🟢 gestures: long-press, then swipe left 9.3s
🟢 gestures: double-tap completes the flow 10.0s
🟢 sticky chrome: accept below the fold, then continue 12.8s
🟢 async states: load, pull to refresh, claim through the toast 12.7s
🟢 huge virtualized list reaches row 512 1m 59s
🟢 tests/flows.e2e.ts · 7 passed 49.8s
🟢 text inputs unlock the submit button 6.4s
🟢 filled text inputs report their values 5.6s
🟢 modal flow confirms through the native alert 8.6s
🟢 bottom tabs act inside the last tab 6.3s
🟢 error recovery retries, then confirms the delete 8.5s
🟢 choice controls place the exact order 5.8s
🟢 debounced search selects the target once results arrive 8.5s
🟢 tests/login-form.e2e.ts · 3 passed 27.8s
🟢 login form › rejects a malformed email 7.4s
🟢 login form › rejects an unknown account 7.3s
🟢 login form › signs in and logs out 13.1s
🟢 tests/smoke.e2e.ts · 1 passed 6.9s
🟢 home lists the scenarios and opens one 6.9s

Tested by e2e 0.23.0 · 10m 6s · android-emulator · run artifacts

@okwasniewski

Copy link
Copy Markdown
Member Author

[claude-opus-5-5] responding on behalf of okwasniewski

Left for a human (also under "For review" in the description):

  • Decision: with no handler, an agent step's confirm is accepted. The agent can't see it, so a destructive confirm goes through. The safer alternative (dismiss, tell the model, let it accept explicitly) needs a model-visible notice first.
  • New engine contract field: optional OperationContext.agentStep (wire, client, server, engine.mdx). Older cores omit it and the engine falls back to origin, so the web peer floor is unchanged.
  • Follow-up, not done: answered dialogs are visible only in the app log, not to the model or to MCP callers.
  • feat(web)!: WebDriver BiDi driver replaces Playwright #1503 will need a small port: driver/dialogs.ts conflicts only in imports, and BidiSession would pass the same two callbacks.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove this file

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 24 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/web-benchmark/tests-agent/scenarios.e2e.ts">

<violation number="1" location="apps/web-benchmark/tests-agent/scenarios.e2e.ts:135">
P2: This scenario has no committed recording under `apps/web-benchmark/.e2e/cache/`. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with `test:agent` and commit the new cache entry in this PR.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/reference/engine.mdx Resolved
},
{
// The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included.
slug: 'deferred-dialog',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This scenario has no committed recording under apps/web-benchmark/.e2e/cache/. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with test:agent and commit the new cache entry in this PR.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/web-benchmark/tests-agent/scenarios.e2e.ts, line 135:

<comment>This scenario has no committed recording under `apps/web-benchmark/.e2e/cache/`. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with `test:agent` and commit the new cache entry in this PR.</comment>

<file context>
@@ -130,6 +130,12 @@ const SCENARIOS: readonly Scenario[] = [
   },
+  {
+    // The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included.
+    slug: 'deferred-dialog',
+    goal: 'submit the report and accept the confirmation that follows',
+    success: 'Report submitted after deferred confirmation',
</file context>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants