Skip to content

feat(web)!: WebDriver BiDi driver replaces Playwright - #1503

Open
okwasniewski wants to merge 1 commit into
mainfrom
oskar/webdriver-bidi
Open

okwasniewski wants to merge 1 commit into
mainfrom
oskar/webdriver-bidi

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Summary

What breaks is in .changeset/web-bidi-only.md. Limits are in docs/reference/web.mdx.

Benchmark totals (before the rebase; 20 warm / 10 cold iterations, M5 Pro, headless)

Total PW chromium BiDi chromium PW firefox BiDi firefox
sum of p50s, every scenario (ms) 1534 981 (-36%) 2623 2116 (-19%)
cold launch + new page + 6 agent steps (ms) 334 274 (-18%) 768 1279 (+66%)
testbed wall clock, median of 3 (ms) 17525 16895 (-4%) 21102 20980 (-1%)
node + browser CPU, warm loop (s) 51.7 20.3 (-61%) 77.7 33.7 (-57%)
peak RSS, node + browser (MB) 1405 1542 (+10%) 2351 2644 (+12%)

Firefox cold launch is slower: the Remote Agent holds session.new until startup finishes (925 ms vs 314 ms), once per worker.

Issue

Spike, no issue.

Verified

Ran it locally: yes

  • @e2e-dev/web: 1483 tests on Chromium and Firefox, 0 failures. One env-gated skip: the crosscheck benchmark scenarios, which run in benchmark.yml.
  • Testbed through the built CLI on Chromium, and on Firefox without the Playwright-tagged tests: all pass.
  • e2e 4718 and core tests pass. Build, typecheck, oxlint, fallow, docs, plugin, MCP, peer-range, and changeset checks are clean.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

@github-actions github-actions Bot added the size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 93 passed

All 93 tests in 42 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 923ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 754ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 169ms
🟢 tests/async-states.e2e.ts · 2 passed 6.9s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.7s
🟢 tests/canvas-only.e2e.ts · 2 passed 329ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 173ms
🟢 canvas only › a shape out of order paints the error and resets progress 156ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 611ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 371ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 240ms
🟢 tests/cart-totals.e2e.ts · 2 passed 405ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 224ms
🟢 cart totals › a quantity never drops below one 181ms
🟢 tests/checkout-review.e2e.ts · 1 passed 181ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 181ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 675ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 348ms
🟢 context menu trap › the decoy menu items refuse and keep the file 327ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.7s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 296ms
🟢 control inventory › check picks a radio the pick replaces with its summary 172ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.0s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 284ms
🟢 control inventory › setInputFiles attaches files named from the project root 231ms
🟢 control inventory › the details view is left through the browser history 207ms
🟢 control inventory › scrollIntoView brings the footnote into view 159ms
🟢 control inventory › every exercise done shows the success message 1.4s
🟢 tests/css-content-ui.e2e.ts · 2 passed 441ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 252ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 189ms
🟢 tests/date-picker.e2e.ts · 2 passed 767ms
🟢 date picker › navigates three months ahead and books the required date 438ms
🟢 date picker › confirming without the exact date shows the error 329ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 649ms
🟢 div soup › a wrong code is rejected 296ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 353ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 615ms
🟢 drag and drop › an out-of-order item resets the dropzone 301ms
🟢 drag and drop › dropping Banana then Cherry passes 314ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 283ms
🟢 filter deep link › filtering writes the query to the URL 150ms
🟢 filter deep link › opening a shared link restores the filter 133ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 596ms
🟢 gift card purchase › refuses to buy without an amount 163ms
🟢 gift card purchase › refuses a recipient without an at sign 228ms
🟢 gift card purchase › sends the gift card 205ms
🟢 tests/hover-menu.e2e.ts · 2 passed 475ms
🟢 hover menu › a decoy leaf is refused 223ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 252ms
🟢 tests/iframe-form.e2e.ts · 2 passed 460ms
🟢 iframe form › a wrong code is refused by the host page 225ms
🟢 iframe form › the code read from one frame applies through the other 235ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 558ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 146ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 177ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 235ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.8s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.8s
🟢 tests/kanban-board.e2e.ts · 3 passed 833ms
🟢 kanban board › pointer drags move both cards into the goal layout 441ms
🟢 kanban board › a swipe along a path drags a card as well 209ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 183ms
🟢 tests/login-form.e2e.ts · 4 passed 836ms
🟢 login form › rejects a malformed email 192ms
🟢 login form › rejects a short password 206ms
🟢 login form › rejects an unknown account 222ms
🟢 login form › signs in and logs out 216ms
🟢 tests/lying-labels.e2e.ts · 4 passed 778ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 166ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 134ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 242ms
🟢 lying labels › visible text drives the payment through 236ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 492ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 165ms
🟢 native dialogs › dismissing the confirm cancels the order 159ms
🟢 native dialogs › a wrong gift code is rejected 168ms
🟢 tests/new-tab-flow.e2e.ts · 2 passed 468ms
🟢 new tab flow › reads the code in the tab the app opens and enters it back in the original tab 228ms
🟢 new tab flow › a tap that opens a tab leaves the screen on the original tab, and activate shows the new one 240ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.4s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 280ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 317ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.3s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.5s
🟢 tests/order-history.e2e.ts · 2 passed 1.3s
🟢 order history › overview and settings render their panels 176ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.3s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 445ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 146ms
🟢 otp auto-advance › a wrong code clears every box and starts over 744ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 677ms
🟢 continues only after the banner and the overlay are gone 677ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 403ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 209ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 194ms
🟢 tests/price-sorting.e2e.ts · 3 passed 312ms
🟢 price sorting › opens in the featured order with sale prices struck through 98ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 104ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 110ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 510ms
🟢 rich text editor › bolding exactly the required word passes the check 257ms
🟢 rich text editor › bold that spills into the previous word fails the check 253ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 476ms
🟢 saved addresses › adds an address and clears the form 167ms
🟢 saved addresses › ignores an add with an empty field 146ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 163ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 320ms
🟢 shadow DOM form › rejects a wrong access code 156ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 164ms
🟢 tests/smoke.e2e.ts · 1 passed 261ms
🟢 home lists the scenarios and opens one 261ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 316ms
🟢 accepts the terms once the button clears the sticky footer 316ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 21.8s
🟢 scrolls the windowed table to the Golden Row and claims it 21.8s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 237ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.2s

Tested by e2e 0.23.1 · 1m 10s · web · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 25 model calls · 119.5k tokens (94% cached) · $0.0048

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.3s
🟢 control inventory › check sets the checkbox and the radio 1 step 655ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 243ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 890ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.5s
🟢 control inventory › upload attaches the named project files 1 step 2.4s
🟢 control inventory › back returns from the details view to the inventory 1 step 276ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 254ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 343ms
🟢 act signs in with the benchmark account 1 step 343ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 13.9s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 7.4s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.3s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 5.2s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 17 calls 1m 27s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.7s
🟢 act completes lying-labels 1 step · 3 calls 5.1s
🟢 act completes iframe-form 1 step · 3 calls 4.7s
🟢 act completes infinite-scroll 1 step 3.8s
🟢 act completes virtualized-table 1 step 26.2s
🟢 act completes overlay-trap 1 step 1.0s
🟢 act completes sticky-chrome 1 step 1.1s
🟢 act completes async-states 1 step · 3 calls 6.6s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 596ms
🟢 act completes drag-and-drop 1 step 618ms
🟢 act completes kanban-board 1 step 824ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 528ms
🟢 act completes otp-auto-advance 1 step 755ms
🟢 act completes date-picker 1 step 971ms
🟢 act completes rich-text-editor 1 step 10.6s
🟢 act completes playbook-cleanup 1 step 704ms
🟢 act completes promo-storefront 1 step 1.4s
🟢 act completes new-tab-flow 1 step · 5 calls 8.7s
🟢 act completes gift-card-purchase 1 step 2.5s
🟢 act filters the list and the URL carries the filter 1 step 271ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 288ms
🟢 act adds an address with a run-unique label 1 step 288ms

Tested by e2e 0.23.1 · 1m 49s · web · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 41 model calls · 233.9k tokens (84% cached) · $0.0150

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 41 calls 6m 57s
🟢 act completes Login Form 1 step 36.5s
🟢 act completes Infinite Scroll List 1 step 25.7s
🟢 act completes Modal Flow 1 step 13.9s
🟢 act completes Bottom Tabs 1 step 9.9s
🟢 act completes Text Input Variations 1 step 9.5s
🟢 act completes Broken Accessibility 2 steps · 5 calls 17.0s
🟢 act completes Vision Only 2 steps · 1 call 7.4s
🟢 act completes Gestures 1 step · 9 calls 25.9s
🟢 act completes Async States 1 step · 13 calls 30.8s
🟢 act completes Debounced Search 1 step 7.6s
🟢 act completes Huge Virtualized List 1 step 1m 53s
🟢 act completes Flattened Registration Form 2 steps · 1 call 11.5s
🟢 act completes Flattened Login 2 steps · 7 calls 22.5s
🟢 act completes Sticky Chrome Target 1 step 10.6s
🟢 act completes WebView Accessibility 1 step · 5 calls 27.4s
🟢 act completes Permission Prompt 1 step 7.9s
🟢 act completes Product Catalog 1 step 9.5s
🟢 act completes Error Recovery 1 step 10.1s
🟢 act completes Choice Controls 1 step 9.4s
🟢 act completes Sequential Onboarding 1 step 10.8s

Tested by e2e 0.23.1 · 6m 58s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 58 passed, 1 skipped

All 59 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 1s
🟢 control inventory › a label query resolves a labeled text field 24.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 13.4s
🟢 control inventory › a placeholder query and attribute reads find the field 9.4s
🟢 control inventory › a secure field takes a fill and denies value reads 9.2s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.0s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.5s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.2s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.3s
🟢 control inventory › a tab role answers getByRole 3.9s
🟢 control inventory › a radio group reports the checked option 7.6s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.0s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.4s
🟢 control inventory › a header answers its accessible name 3.8s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.5s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.4s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.2s
🟢 control inventory › a double tap lands inside the 300 ms window 6.6s
🟢 control inventory › an ambiguous locator fails at once 4.9s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.4s
🟢 refusals › verbs a device never declares are refused before the engine 3ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.0s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 14s
🟢 device fixture › setOrientation rotates the window the app measures 4.8s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.0s
🟢 device fixture › foregroundApp sees the home screen after home() 3.3s
🟢 device fixture › closeApp then openApp relaunches on the home list 5.6s
🟢 device fixture › closeApp leaves the next commands on the same device 5.8s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.0s
🟢 device fixture › openLink opens the scenario a deep link names 4.3s
🟢 device fixture › device.back pops the scenario 3.9s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 3.3s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 14.5s
🟢 device fixture › setLocation and clearLocation change what the app reads 20.4s
🟢 tests/device.e2e.ts · 5 passed 46.7s
🟢 permission prompt › allowing the system dialog enables the microphone 6.7s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 18.2s
🟢 sequential onboarding echoes the exact values 9.4s
🟢 onboarding rejects a malformed email before advancing 6.0s
🟢 dismissing the keyboard over choice controls leaves them as they were 6.5s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 9s
🟢 infinite scroll reaches the target item 27.6s
🟢 product catalog adds exactly two of the right variant 8.3s
🟢 gestures: long-press, then swipe left 7.8s
🟢 gestures: double-tap completes the flow 9.6s
🟢 sticky chrome: accept below the fold, then continue 10.6s
🟢 async states: load, pull to refresh, claim through the toast 11.5s
🟢 huge virtualized list reaches row 512 1m 54s
🟢 tests/flows.e2e.ts · 7 passed 1m 0s
🟢 text inputs unlock the submit button 9.4s
🟢 filled text inputs report their values 6.7s
🟢 modal flow confirms through the native alert 10.6s
🟢 bottom tabs act inside the last tab 8.5s
🟢 error recovery retries, then confirms the delete 9.1s
🟢 choice controls place the exact order 6.7s
🟢 debounced search selects the target once results arrive 9.0s
🟢 tests/login-form.e2e.ts · 3 passed 29.9s
🟢 login form › rejects a malformed email 7.8s
🟢 login form › rejects an unknown account 7.8s
🟢 login form › signs in and logs out 14.3s
🟢 tests/smoke.e2e.ts · 1 passed 6.7s
🟢 home lists the scenarios and opens one 6.7s

Tested by e2e 0.23.1 · 9m 49s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 56 passed, 1 skipped

All 57 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 2m 57s
🟢 control inventory › a label query resolves a labeled text field 13.3s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 15.3s
🟢 control inventory › a placeholder query and attribute reads find the field 5.6s
🟢 control inventory › a secure field takes a fill and denies value reads 7.4s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.0s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.1s
🟢 control inventory › toBeFocused follows the focus 9.0s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.3s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.9s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.5s
🟢 control inventory › a radio group reports the checked option 9.1s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.6s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.1s
🟢 control inventory › a header answers its accessible name 4.0s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.1s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.5s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.8s
🟢 control inventory › pointer verbs land at the point asked for 9.0s
🟢 control inventory › an ambiguous locator fails at once 5.6s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.6s
🟢 refusals › verbs a device never declares are refused before the engine 1ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 588ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 12 passed 1m 51s
🟢 device fixture › setAppearance flips the color scheme the app reads 16.6s
🟢 device fixture › setOrientation rotates the window the app measures 9.5s
🟢 device fixture › the clipboard reads back what was written 7.2s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.4s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.1s
🟢 device fixture › closeApp leaves the next commands on the same device 12.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.8s
🟢 device fixture › openLink opens the scenario a deep link names 13.0s
🟢 device fixture › device.back pops the scenario 6.9s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.6s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.4s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.2s
🟢 tests/device.e2e.ts · 3 passed 47.0s
🟢 sequential onboarding echoes the exact values 26.0s
🟢 onboarding rejects a malformed email before advancing 10.1s
🟢 dismissing the keyboard over choice controls leaves them as they were 10.8s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 5s
🟢 infinite scroll reaches the target item 34.0s
🟢 product catalog adds exactly two of the right variant 10.4s
🟢 gestures: long-press, then swipe left 8.9s
🟢 sticky chrome: accept below the fold, then continue 12.4s
🟢 async states: load, pull to refresh, claim through the toast 12.4s
🟢 huge virtualized list reaches row 512 2m 34s
🟢 web view coupon form applies the code shown on the page 12.5s
🟢 tests/flows.e2e.ts · 7 passed 1m 8s
🟢 text inputs unlock the submit button 11.8s
🟢 filled text inputs report their values 10.5s
🟢 modal flow confirms through the native alert 9.7s
🟢 bottom tabs act inside the last tab 7.4s
🟢 error recovery retries, then confirms the delete 9.3s
🟢 choice controls place the exact order 7.4s
🟢 debounced search selects the target once results arrive 12.0s
🟢 tests/login-form.e2e.ts · 3 passed 45.3s
🟢 login form › rejects a malformed email 13.1s
🟢 login form › rejects an unknown account 13.0s
🟢 login form › signs in and logs out 19.2s
🟢 tests/smoke.e2e.ts · 1 passed 8.8s
🟢 home lists the scenarios and opens one 8.8s

Tested by e2e 0.23.1 · 6m 43s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 55 model calls · 337.0k tokens (82% cached) · $0.0224

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 55 calls 8m 13s
🟢 act completes Login Form 1 step 35.0s
🟢 act completes Infinite Scroll List 1 step 36.7s
🟢 act completes Modal Flow 1 step 9.9s
🟢 act completes Bottom Tabs 1 step 7.3s
🟢 act completes Text Input Variations 1 step 12.4s
🟢 act completes Broken Accessibility 2 steps · 6 calls 27.6s
🟢 act completes Vision Only 2 steps · 6 calls 28.0s
🟢 act completes Gestures 1 step · 9 calls 39.6s
🟢 act completes Async States 1 step · 7 calls 23.1s
🟢 act completes Debounced Search 1 step · 1 call 27.7s
🟢 act completes Flattened Registration Form 2 steps · 1 call 25.2s
🟢 act completes Flattened Login 2 steps · 10 calls 54.1s
🟢 act completes Sticky Chrome Target 1 step 12.9s
🟢 act completes WebView Accessibility 1 step 11.9s
🟢 act completes Permission Prompt 1 step 11.0s
🟢 act completes Photo Picker 1 step · 2 calls 29.3s
🟢 act completes Product Catalog 1 step 10.0s
🟢 act completes Error Recovery 1 step 9.4s
🟢 act completes Choice Controls 1 step · 6 calls 39.1s
🟢 act completes Stripe PaymentSheet 1 step · 7 calls 31.9s
🟢 act completes Apple Pay 1 step 10.6s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.1 · 8m 13s · ios-simulator · run artifacts

@okwasniewski
okwasniewski changed the base branch from main to oskar/playwright-parity-tests November 1, 2026 18:40
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 2 times, most recently from 2411751 to 7e2fbc6 Compare November 1, 2026 19:12
@okwasniewski
okwasniewski force-pushed the oskar/playwright-parity-tests branch 2 times, most recently from b9422fc to 22d080f Compare November 4, 2026 16:57
Base automatically changed from oskar/playwright-parity-tests to main November 4, 2026 18:02
@okwasniewski okwasniewski changed the title feat(web): experimental WebDriver BiDi driver feat(web)!: WebDriver BiDi driver replaces Playwright Nov 6, 2026
Comment thread packages/web/src/install/download.ts Dismissed
@okwasniewski
okwasniewski marked this pull request as ready for review November 16, 2026 17:20
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 7 times, most recently from ff9683d to 34d49bb Compare November 16, 2026 17:57
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 11 times, most recently from 29b7cfc to 7120c38 Compare November 16, 2026 20:33
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 2 times, most recently from ff47822 to f7b38c2 Compare November 16, 2026 21:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants