Skip to content

test(testbed): Playwright's page tests in the testbed suite - #1505

Merged
okwasniewski merged 9 commits into
mainfrom
oskar/playwright-parity-tests
Nov 4, 2026
Merged

okwasniewski merged 9 commits into
mainfrom
oskar/playwright-parity-tests

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

What and why

  • Playwright's own page tests (microsoft/playwright tests/page + tests/library/download.spec.ts at v1.64.1, our playwright-core pin), rewritten against the e2e public API, as ordinary testbed tests in apps/testbed/tests/. 32 files, ~500 tests: click, fill, check, select, keyboard, locators, actionability and scroll into view, frames, dialogs, route/fulfill/continue/fallback, waitForURL/Response, network idle, screenshots + masks, downloads, file chooser.
  • Each describe is tagged playwright (--tag playwright runs only them). Each file header names its source and lists the tests left out, with the reason.
  • Known differences are fail: 'reason' (feat(e2e): test.fail for tests expected to fail #1539), so a closed gap fails the run until its mark goes: 16 tests. 6 fail with an error fail does not cover (an engine failure, or a refused argument), so they are skip: 'reason'.
  • The playground serves Playwright's assets and its test server's dynamic routes for any path it has no page for (apps/testbed/app/playwright/, with LICENSE + NOTICE; assets unchanged, ignored by lint and fallow). One app, one config, chromium only.
  • CI: on a pull request the Testbed step leaves the playwright tests out of the model-filtered run and a second step runs all of them (23s locally); a push to main runs everything once.
  • 6 testbed tests the ported ones cover are removed (custom dialog handler, route stub/abort, raw keyboard, scrollIntoView, waitForURL).
  • One engine fix the tests found: a fixture call that times out names what it waited for (operation timed out in browser.waitForResponse "foo.css"), and url-pattern steps quote string patterns.

Marked fail (open gaps unless noted):

  • elements inside a visibility: hidden iframe read as visible (3 tests)
  • uncheck refuses role=radio; inputValue on a non-input reads "" instead of throwing
  • route.fulfill body string-only; header overrides can't delete a header
  • ? in route globs matches one character (Playwright: literal)
  • a page deleting Map/Array breaks evaluate (main world)
  • two screenshot steps at once interfere
  • by design per the migrate guide: bare-string selectOption matches label only, textContent() is rendered text, isChecked() on a non-checkable reads false

Marked skip: multiple set-cookie in a fulfill (2), screenshot while navigating, screenshot with a stalled subframe, setInputFiles([]), timeout: 0.

CodeQL on the routes (apps/testbed/app/playwright/routes.mjs, test-only): the app listens on 127.0.0.1 only; delays and sizes are clamped, redirects stay on the app's two origins, credentialed CORS goes only to those origins, the multipart parse is linear.

Issue

None. #1503 stacks on this.

Verified

Ran it locally: yes

apps/testbed, chromium (macOS), on main 732ea8af9
e2e run --tag playwright            503 passed | 6 skipped      23s
pnpm test --exclude-tag playwright   76 passed | 1 skipped
pnpm test (all)                     579 passed | 7 skipped      29s
e2e + web unit, lint, fallow, typecheck, build clean

Checklist

  • Changeset added for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change.
  • Docs updated in the same PR (error text only; no page describes it).
  • Contract change.
  • Engine contract change.

@github-actions github-actions Bot added the size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 766ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 546ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 220ms
🟢 tests/async-states.e2e.ts · 2 passed 7.2s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.3s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 514ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 265ms
🟢 canvas only › a shape out of order paints the error and resets progress 249ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 710ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 382ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 328ms
🟢 tests/cart-totals.e2e.ts · 2 passed 634ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 362ms
🟢 cart totals › a quantity never drops below one 272ms
🟢 tests/checkout-review.e2e.ts · 1 passed 275ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 275ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 899ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 488ms
🟢 context menu trap › the decoy menu items refuse and keep the file 411ms
🟢 tests/control-inventory.e2e.ts · 8 passed 4.6s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 491ms
🟢 control inventory › check picks a radio the pick replaces with its summary 291ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.2s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 357ms
🟢 control inventory › setInputFiles attaches files named from the project root 250ms
🟢 control inventory › the details view is left through the browser history 274ms
🟢 control inventory › scrollIntoView brings the footnote into view 213ms
🟢 control inventory › every exercise done shows the success message 1.5s
🟢 tests/css-content-ui.e2e.ts · 2 passed 572ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 323ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 249ms
🟢 tests/date-picker.e2e.ts · 2 passed 835ms
🟢 date picker › navigates three months ahead and books the required date 481ms
🟢 date picker › confirming without the exact date shows the error 354ms
🟢 tests/debounced-search.e2e.ts · 2 passed 3.0s
🟢 debounced search › a decoy one character off is refused 1.5s
🟢 debounced search › picking the exact item adds it to the cart 1.5s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.2s
🟢 one submit opens one confirm, even when double-clicked 1.2s
🟢 tests/div-soup.e2e.ts · 2 passed 939ms
🟢 div soup › a wrong code is rejected 422ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 517ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 791ms
🟢 drag and drop › an out-of-order item resets the dropzone 397ms
🟢 drag and drop › dropping Banana then Cherry passes 394ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 343ms
🟢 filter deep link › filtering writes the query to the URL 201ms
🟢 filter deep link › opening a shared link restores the filter 142ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 789ms
🟢 gift card purchase › refuses to buy without an amount 204ms
🟢 gift card purchase › refuses a recipient without an at sign 283ms
🟢 gift card purchase › sends the gift card 302ms
🟢 tests/hover-menu.e2e.ts · 2 passed 622ms
🟢 hover menu › a decoy leaf is refused 279ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 343ms
🟢 tests/iframe-form.e2e.ts · 2 passed 710ms
🟢 iframe form › a wrong code is refused by the host page 336ms
🟢 iframe form › the code read from one frame applies through the other 374ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 563ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 139ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 194ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 230ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 3.0s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 3.0s
🟢 tests/kanban-board.e2e.ts · 3 passed 794ms
🟢 kanban board › pointer drags move both cards into the goal layout 393ms
🟢 kanban board › a swipe along a path drags a card as well 179ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 222ms
🟢 tests/login-form.e2e.ts · 4 passed 859ms
🟢 login form › rejects a malformed email 209ms
🟢 login form › rejects a short password 198ms
🟢 login form › rejects an unknown account 201ms
🟢 login form › signs in and logs out 251ms
🟢 tests/lying-labels.e2e.ts · 4 passed 728ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 145ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 108ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 257ms
🟢 lying labels › visible text drives the payment through 218ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 708ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 212ms
🟢 native dialogs › dismissing the confirm cancels the order 210ms
🟢 native dialogs › a wrong gift code is rejected 286ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.4s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.4s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 6.8s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 421ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 570ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.7s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 2.0s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 2.1s
🟢 tests/order-history.e2e.ts · 2 passed 1.6s
🟢 order history › overview and settings render their panels 281ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.3s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.7s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 551ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 223ms
🟢 otp auto-advance › a wrong code clears every box and starts over 953ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 773ms
🟢 continues only after the banner and the overlay are gone 773ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 714ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 366ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 348ms
🟢 tests/price-sorting.e2e.ts · 3 passed 614ms
🟢 price sorting › opens in the featured order with sale prices struck through 171ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 180ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 263ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.3s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.3s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 659ms
🟢 rich text editor › bolding exactly the required word passes the check 342ms
🟢 rich text editor › bold that spills into the previous word fails the check 317ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 696ms
🟢 saved addresses › adds an address and clears the form 242ms
🟢 saved addresses › ignores an add with an empty field 210ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 244ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 487ms
🟢 shadow DOM form › rejects a wrong access code 236ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 251ms
🟢 tests/smoke.e2e.ts · 1 passed 470ms
🟢 home lists the scenarios and opens one 470ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 278ms
🟢 accepts the terms once the button clears the sticky footer 278ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 22.0s
🟢 scrolls the windowed table to the Golden Row and claims it 22.0s
🟢 tests/wishlist.e2e.ts · 2 passed 1.6s
🟢 wishlist › saving toggles the button and the tab badge 290ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.22.0 · 1m 19s · web · run artifacts

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 33 passed

34 agent steps · 26 replayed from cache · 20 model calls · 87.6k tokens (85% cached) · $0.0053

All 33 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 9.9s
🟢 control inventory › check sets the checkbox and the radio 1 step 1.1s
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 572ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 1.4s
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 3.0s
🟢 control inventory › upload attaches the named project files 1 step 2.8s
🟢 control inventory › back returns from the details view to the inventory 1 step 536ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 471ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 651ms
🟢 act signs in with the benchmark account 1 step 651ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 16.5s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 8.7s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 2.2s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 5.6s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 12 calls 2m 35s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.2s
🟢 act completes lying-labels 1 step · 3 calls 5.3s
🟢 act completes iframe-form 1 step · 3 calls 5.2s
🟢 act completes infinite-scroll 1 step 7.9s
🟢 act completes virtualized-table 1 step 1m 33s
🟢 act completes overlay-trap 1 step 1.0s
🟢 act completes sticky-chrome 1 step 1.2s
🟢 act completes async-states 1 step · 3 calls 8.8s
🟢 act completes debounced-search 1 step 2.7s
🟢 act completes hover-menu 1 step 779ms
🟢 act completes drag-and-drop 1 step 656ms
🟢 act completes kanban-board 1 step 927ms
🟢 act completes context-menu-trap 1 step 2.9s
🟢 act completes stale-dom 1 step 713ms
🟢 act completes otp-auto-advance 1 step 1.0s
🟢 act completes date-picker 1 step 1.3s
🟢 act completes rich-text-editor 1 step 11.6s
🟢 act completes playbook-cleanup 1 step 1.0s
🟢 act completes promo-storefront 1 step 1.6s
🟢 act completes gift-card-purchase 1 step 2.7s
🟢 act filters the list and the URL carries the filter 1 step 416ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 635ms
🟢 act adds an address with a run-unique label 1 step 635ms

Tested by e2e 0.22.0 · 3m 4s · web · run artifacts

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 40 model calls · 232.8k tokens (79% cached) · $0.0172

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 40 calls 5m 52s
🟢 act completes Login Form 1 step 25.5s
🟢 act completes Infinite Scroll List 1 step 23.2s
🟢 act completes Modal Flow 1 step 11.7s
🟢 act completes Bottom Tabs 1 step 7.6s
🟢 act completes Text Input Variations 1 step 7.0s
🟢 act completes Broken Accessibility 2 steps · 6 calls 21.7s
🟢 act completes Vision Only 2 steps · 1 call 8.0s
🟢 act completes Gestures 1 step · 7 calls 22.7s
🟢 act completes Async States 1 step · 13 calls 36.6s
🟢 act completes Debounced Search 1 step 15.3s
🟢 act completes Huge Virtualized List 1 step 1m 12s
🟢 act completes Flattened Registration Form 2 steps · 1 call 9.7s
🟢 act completes Flattened Login 2 steps · 6 calls 17.8s
🟢 act completes Sticky Chrome Target 1 step 7.7s
🟢 act completes WebView Accessibility 1 step · 6 calls 20.3s
🟢 act completes Permission Prompt 1 step 7.4s
🟢 act completes Product Catalog 1 step 9.1s
🟢 act completes Error Recovery 1 step 10.0s
🟢 act completes Choice Controls 1 step 9.3s
🟢 act completes Sequential Onboarding 1 step 10.0s

Tested by e2e 0.22.0 · 5m 53s · android-emulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 122 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/web/tests/parity/server.mjs Resolved
Comment thread packages/web/tests/parity/server.mjs Resolved
Comment thread apps/testbed/tests/download.e2e.ts Resolved
Comment thread packages/web/tests/parity/e2e.config.ts Resolved
Comment thread apps/testbed/tests/click.e2e.ts Resolved
Comment thread packages/web/tests/parity/server.mjs Resolved
Comment thread packages/web/tests/parity/server.mjs Resolved
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 9s
🟢 control inventory › a label query resolves a labeled text field 28.3s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 11.4s
🟢 control inventory › a placeholder query and attribute reads find the field 5.2s
🟢 control inventory › a secure field takes a fill and denies value reads 8.9s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.3s
🟢 control inventory › toBeFocused follows the focus 10.6s
🟢 control inventory › a switch checks, unchecks, and reports its state 8.8s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.4s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 10.6s
🟢 control inventory › a tab role answers getByRole 4.0s
🟢 control inventory › a radio group reports the checked option 8.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.9s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.0s
🟢 control inventory › a header answers its accessible name 4.1s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.8s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.6s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.1s
🟢 control inventory › a double tap lands inside the 300 ms window 7.1s
🟢 control inventory › an ambiguous locator fails at once 5.9s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 13.0s
🟢 refusals › verbs a device never declares are refused before the engine 11ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 955ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.2s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 40s
🟢 device fixture › setOrientation rotates the window the app measures 6.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.1s
🟢 device fixture › foregroundApp sees the home screen after home() 5.1s
🟢 device fixture › closeApp then openApp relaunches on the home list 10.5s
🟢 device fixture › closeApp leaves the next commands on the same device 8.7s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.0s
🟢 device fixture › device.back pops the scenario 6.7s
🟢 device fixture › unstable_raw reaches the test session through agent-device 5.7s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 19.1s
🟢 device fixture › setLocation and clearLocation change what the app reads 26.0s
🟢 tests/device.e2e.ts · 5 passed 49.3s
🟢 permission prompt › allowing the system dialog enables the microphone 6.5s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 14.0s
🟢 sequential onboarding echoes the exact values 11.9s
🟢 onboarding rejects a malformed email before advancing 7.8s
🟢 dismissing the keyboard over choice controls leaves them as they were 9.1s
🟢 tests/flows-lists.e2e.ts · 7 passed 2m 7s
🟢 infinite scroll reaches the target item 21.3s
🟢 product catalog adds exactly two of the right variant 8.9s
🟢 gestures: long-press, then swipe left 8.1s
🟢 gestures: double-tap completes the flow 9.5s
🟢 sticky chrome: accept below the fold, then continue 8.7s
🟢 async states: load, pull to refresh, claim through the toast 12.5s
🟢 huge virtualized list reaches row 512 57.9s
🟢 tests/flows.e2e.ts · 7 passed 53.5s
🟢 text inputs unlock the submit button 6.2s
🟢 filled text inputs report their values 5.7s
🟢 modal flow confirms through the native alert 9.6s
🟢 bottom tabs act inside the last tab 7.0s
🟢 error recovery retries, then confirms the delete 9.1s
🟢 choice controls place the exact order 6.3s
🟢 debounced search selects the target once results arrive 9.6s
🟢 tests/login-form.e2e.ts · 3 passed 29.7s
🟢 login form › rejects a malformed email 8.9s
🟢 login form › rejects an unknown account 8.1s
🟢 login form › signs in and logs out 12.7s
🟢 tests/smoke.e2e.ts · 1 passed 6.4s
🟢 home lists the scenarios and opens one 6.4s

Tested by e2e 0.22.0 · 9m 16s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 3s
🟢 control inventory › a label query resolves a labeled text field 10.7s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.9s
🟢 control inventory › a placeholder query and attribute reads find the field 4.4s
🟢 control inventory › a secure field takes a fill and denies value reads 8.9s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.5s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.9s
🟢 control inventory › toBeFocused follows the focus 10.3s
🟢 control inventory › a switch checks, unchecks, and reports its state 11.9s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.4s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.9s
🟢 control inventory › a radio group reports the checked option 9.1s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.8s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.2s
🟢 control inventory › a header answers its accessible name 4.2s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.3s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.4s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.8s
🟢 control inventory › pointer verbs land at the point asked for 9.6s
🟢 control inventory › an ambiguous locator fails at once 5.7s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 11.8s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 565ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 41s
🟢 device fixture › setAppearance flips the color scheme the app reads 13.8s
🟢 device fixture › setOrientation rotates the window the app measures 8.3s
🟢 device fixture › the clipboard reads back what was written 6.3s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.2s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.0s
🟢 device fixture › closeApp leaves the next commands on the same device 11.7s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 10.3s
🟢 device fixture › device.back pops the scenario 8.7s
🟢 device fixture › unstable_raw reaches the test session through agent-device 5.9s
🟢 device fixture › setLocation and clearLocation change what the app reads 11.1s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.4s
🟢 tests/device.e2e.ts · 3 passed 1m 7s
🟢 sequential onboarding echoes the exact values 41.2s
🟢 onboarding rejects a malformed email before advancing 14.7s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.5s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 16s
🟢 infinite scroll reaches the target item 24.3s
🟢 product catalog adds exactly two of the right variant 10.0s
🟢 gestures: long-press, then swipe left 9.0s
🟢 sticky chrome: accept below the fold, then continue 10.6s
🟢 async states: load, pull to refresh, claim through the toast 10.8s
🟢 huge virtualized list reaches row 512 2m 0s
🟢 web view coupon form applies the code shown on the page 12.1s
🟢 tests/flows.e2e.ts · 7 passed 1m 10s
🟢 text inputs unlock the submit button 13.1s
🟢 filled text inputs report their values 10.3s
🟢 modal flow confirms through the native alert 9.7s
🟢 bottom tabs act inside the last tab 7.6s
🟢 error recovery retries, then confirms the delete 9.5s
🟢 choice controls place the exact order 8.5s
🟢 debounced search selects the target once results arrive 11.6s
🟢 tests/login-form.e2e.ts · 3 passed 50.2s
🟢 login form › rejects a malformed email 14.2s
🟢 login form › rejects an unknown account 14.7s
🟢 login form › signs in and logs out 21.3s
🟢 tests/smoke.e2e.ts · 1 passed 8.6s
🟢 home lists the scenarios and opens one 8.6s

Tested by e2e 0.22.0 · 6m 5s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 12 replayed from cache · 62 model calls · 472.0k tokens (81% cached) · $0.0340

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:266

ASSERTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 1m 13s and 13 model calls

agent.act failed: The long-press and left swipe advanced the scenario to Step 3. Double-tapped the “Double-tap me” target, but the screen remained on Step 3 with the same instruction and target, so completion was not verified.

  • Turn 10: tap({"target":"n4403"}) → [tap] Tapped #n4403.
  • Turn 11: tap({"target":"n4448"}) → [tap] Tapped #n4448.
  • Turn 12: double_tap({"target":"n4484"}) → [double_tap] Double-tapped #n4484.
  • Screen: dev.e2e.benchmark / Gestures

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 62 calls 9m 0s
🟢 act completes Login Form 1 step 41.0s
🟢 act completes Infinite Scroll List 1 step 30.8s
🟢 act completes Modal Flow 1 step 10.5s
🟢 act completes Bottom Tabs 1 step 7.9s
🟢 act completes Text Input Variations 1 step 14.9s
🟢 act completes Broken Accessibility 2 steps · 8 calls 43.8s
🟢 act completes Vision Only 2 steps · 6 calls 32.0s
⚠️ act completes Gestures (1 failed attempt first) 1 step · 11 calls 46.2s
🟢 act completes Async States 1 step · 10 calls 36.9s
🟢 act completes Debounced Search 1 step · 1 call 27.4s
🟢 act completes Flattened Registration Form 2 steps · 1 call 29.3s
🟢 act completes Flattened Login 2 steps · 10 calls 54.4s
🟢 act completes Sticky Chrome Target 1 step 10.5s
🟢 act completes WebView Accessibility 1 step 12.2s
🟢 act completes Permission Prompt 1 step 11.5s
🟢 act completes Photo Picker 1 step · 2 calls 30.8s
🟢 act completes Product Catalog 1 step 10.1s
🟢 act completes Error Recovery 1 step 10.2s
🟢 act completes Choice Controls 1 step · 5 calls 33.6s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 34.3s
🟢 act completes Apple Pay 1 step 11.9s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.22.0 · 10m 18s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/web/tests/parity/server.mjs Resolved
@okwasniewski
okwasniewski force-pushed the oskar/playwright-parity-tests branch from 10d6184 to b9422fc Compare November 4, 2026 13:25
@okwasniewski
okwasniewski changed the base branch from main to oskar/test-fail November 4, 2026 13:25
@okwasniewski okwasniewski changed the title test(web): Playwright parity suite on the Playwright driver test(testbed): Playwright's page tests in the testbed suite Nov 4, 2026
@github-actions github-actions Bot added size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile and removed size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile labels Nov 4, 2026
Comment thread apps/testbed/app/playwright/routes.mjs Dismissed
Comment thread apps/testbed/app/playwright/routes.mjs Dismissed
Comment thread apps/testbed/app/playwright/routes.mjs Dismissed
Comment thread apps/testbed/app/playwright/routes.mjs Dismissed
Comment thread apps/testbed/app/playwright/routes.mjs Dismissed
Base automatically changed from oskar/test-fail to main November 4, 2026 15:08
…ns, /ws handshake, pdf type, backslash-safe redirects, cleanup in finally, fixed PARITY_PORT
…ected-gaps.txt

One config, chromium and firefox targets on one server; browserName is a fixture
read from the app URL path. e2eGap skips become tests-parity/expected-gaps.txt,
checked by scripts/parity.ts: unlisted failure, closed gap, stale line, or TODO
fails. Runs in the check job; separate parity job dropped.
Ported specs live in tests/ tagged playwright, served by the playground
(app/playwright/), chromium only. Known gaps are test.fail; four that fail
as engine failures are skipped. Drops the parity config, gap file, script,
and CI step, and 10 testbed tests the ported ones cover.
- gaps refused as INVALID_ARGUMENT move from fail to skip; mask-in-parallel
  awaits both screenshots and names the real failure
- ports about:blank null origin; click.e2e.ts lists the tests left out
- restores testbed tests for onDialog shorthands, frame fill, download.path
- cleans upload temp dirs; caps the slow expected failure at 5s
- CI runs every Playwright test on a pull request; lightpanda and mixed
  leave them out
@okwasniewski
okwasniewski force-pushed the oskar/playwright-parity-tests branch from b9422fc to 22d080f Compare November 4, 2026 16:57
@okwasniewski
okwasniewski merged commit bdf769b into main Nov 4, 2026
36 of 38 checks passed
@okwasniewski
okwasniewski deleted the oskar/playwright-parity-tests branch November 4, 2026 18:02
@github-actions github-actions Bot mentioned this pull request Nov 4, 2026
This was referenced Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants