Skip to content

fix(mobile): bound reads by operation budget with last-read grace - #1464

Merged
okwasniewski merged 8 commits into
mainfrom
oskar/mobile-read-budget
Oct 31, 2026
Merged

okwasniewski merged 8 commits into
mainfrom
oskar/mobile-read-budget

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Takes over #1112 by Totopo27 (closed unmerged after approval), rebased on current main with authorship credited.

  • @e2e-dev/mobile bounds every snapshot and screenshot by the operation's budget (one InputBudget per operation, including relocation and keyboard-dismiss snapshots), so a hung read fails the step instead of holding it until the test timeout.
  • Core's waits run reads through one ReadWindow (packages/e2e/src/internal/time.ts): the last read may run past the deadline by the slowest read the wait has finished, capped by the phase deadline, so a slow last read still finishes and is classed as LOCATOR_NOT_FOUND / ASSERTION_FAILED, not ACTION_FAILED.
  • Reads past the deadline share that extra time instead of each getting it again (toHaveScreenshot with masks makes several reads per sample).
  • A timed-out read is judged by when that read started, not when its sample did, and a negation counts as held only until the deadline.
  • An action whose target resolved past its deadline is never sent: it fails as ACTION_FAILED.
  • Peer floor: mobile requires e2e >=0.22.0 <1. Published e2e is 0.21.3 and a pending e2e: minor changeset (feat(agent)!: keep model screenshots after a secret fill #1437) makes the next release 0.22.0, which ships this. If a release goes out before this merges, the floor needs bumping again.
  • Restored main's expect(calls).toMatchObject({ resolve: 1, read: 2 }) in locator-engine.test.ts (kept).

Issue

Closes #1099

Verified

Ran it locally: yes

  • mobile benchmark, iOS 26.5 sim, e2e run tests/controls.e2e.ts --target ios-simulator, built packages. a missing locator fails as not found once its timeout passes (1 s timeout) is LOCATOR_NOT_FOUND on every run, main and branch.
  • Last branch run: 23 passed, 1 skipped. Earlier runs on this sim were flaky on main too (1 and 4 failures in different tests: long press, pointer verbs, focus), so they are local sim noise, not this change.
   ✓ refusals > a missing locator fails as not found once its timeout passes 1.10s
      Tests  23 passed | 1 skipped (24)

Not covered: a slow last read on a real device. The unit tests cover it, and fail without the grace.

@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 504ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 350ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 154ms
🟢 tests/async-states.e2e.ts · 2 passed 7.0s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.3s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 262ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 142ms
🟢 canvas only › a shape out of order paints the error and resets progress 120ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 378ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 224ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 154ms
🟢 tests/cart-totals.e2e.ts · 2 passed 358ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 210ms
🟢 cart totals › a quantity never drops below one 148ms
🟢 tests/checkout-review.e2e.ts · 1 passed 178ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 178ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 530ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 281ms
🟢 context menu trap › the decoy menu items refuse and keep the file 249ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.7s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 254ms
🟢 control inventory › check picks a radio the pick replaces with its summary 157ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.0s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 284ms
🟢 control inventory › setInputFiles attaches files named from the project root 152ms
🟢 control inventory › the details view is left through the browser history 197ms
🟢 control inventory › scrollIntoView brings the footnote into view 247ms
🟢 control inventory › every exercise done shows the success message 1.4s
🟢 tests/css-content-ui.e2e.ts · 2 passed 311ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 157ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 154ms
🟢 tests/date-picker.e2e.ts · 2 passed 642ms
🟢 date picker › navigates three months ahead and books the required date 377ms
🟢 date picker › confirming without the exact date shows the error 265ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.0s
🟢 one submit opens one confirm, even when double-clicked 1.0s
🟢 tests/div-soup.e2e.ts · 2 passed 597ms
🟢 div soup › a wrong code is rejected 268ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 329ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 647ms
🟢 drag and drop › an out-of-order item resets the dropzone 318ms
🟢 drag and drop › dropping Banana then Cherry passes 329ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 230ms
🟢 filter deep link › filtering writes the query to the URL 128ms
🟢 filter deep link › opening a shared link restores the filter 102ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 500ms
🟢 gift card purchase › refuses to buy without an amount 135ms
🟢 gift card purchase › refuses a recipient without an at sign 180ms
🟢 gift card purchase › sends the gift card 185ms
🟢 tests/hover-menu.e2e.ts · 2 passed 414ms
🟢 hover menu › a decoy leaf is refused 183ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 231ms
🟢 tests/iframe-form.e2e.ts · 2 passed 526ms
🟢 iframe form › a wrong code is refused by the host page 260ms
🟢 iframe form › the code read from one frame applies through the other 266ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 451ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 111ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 144ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 196ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.9s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.9s
🟢 tests/kanban-board.e2e.ts · 3 passed 761ms
🟢 kanban board › pointer drags move both cards into the goal layout 391ms
🟢 kanban board › a swipe along a path drags a card as well 153ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 217ms
🟢 tests/login-form.e2e.ts · 4 passed 761ms
🟢 login form › rejects a malformed email 179ms
🟢 login form › rejects a short password 186ms
🟢 login form › rejects an unknown account 180ms
🟢 login form › signs in and logs out 216ms
🟢 tests/lying-labels.e2e.ts · 4 passed 614ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 136ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 107ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 189ms
🟢 lying labels › visible text drives the payment through 182ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 432ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 169ms
🟢 native dialogs › dismissing the confirm cancels the order 130ms
🟢 native dialogs › a wrong gift code is rejected 133ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.8s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 285ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 334ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.3s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.4s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.5s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 183ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 482ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 148ms
🟢 otp auto-advance › a wrong code clears every box and starts over 776ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 702ms
🟢 continues only after the banner and the overlay are gone 702ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 516ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 283ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 233ms
🟢 tests/price-sorting.e2e.ts · 3 passed 386ms
🟢 price sorting › opens in the featured order with sale prices struck through 111ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 128ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 147ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 440ms
🟢 rich text editor › bolding exactly the required word passes the check 226ms
🟢 rich text editor › bold that spills into the previous word fails the check 214ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 534ms
🟢 saved addresses › adds an address and clears the form 189ms
🟢 saved addresses › ignores an add with an empty field 162ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 183ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 547ms
🟢 shadow DOM form › rejects a wrong access code 263ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 284ms
🟢 tests/smoke.e2e.ts · 1 passed 428ms
🟢 home lists the scenarios and opens one 428ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.1s
🟢 taps the moving target three times across rebuilds of the list 2.1s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 205ms
🟢 accepts the terms once the button clears the sticky footer 205ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 15.0s
🟢 scrolls the windowed table to the Golden Row and claims it 15.0s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 242ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

e2e 0.22.0 · 1m 2s · web · run artifacts

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 33 passed

34 agent steps · 26 replayed from cache · 20 model calls · 85.3k tokens (85% cached) · $0.0052

All 33 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.4s
🟢 control inventory › check sets the checkbox and the radio 1 step 555ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 277ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 935ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.6s
🟢 control inventory › upload attaches the named project files 1 step 2.4s
🟢 control inventory › back returns from the details view to the inventory 1 step 281ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 360ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 373ms
🟢 act signs in with the benchmark account 1 step 373ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 11.7s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 6.0s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.8s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 3.9s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 12 calls 1m 41s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.2s
🟢 act completes lying-labels 1 step · 3 calls 4.6s
🟢 act completes iframe-form 1 step · 3 calls 4.3s
🟢 act completes infinite-scroll 1 step 5.1s
🟢 act completes virtualized-table 1 step 51.2s
🟢 act completes overlay-trap 1 step 844ms
🟢 act completes sticky-chrome 1 step 864ms
🟢 act completes async-states 1 step · 3 calls 6.3s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 464ms
🟢 act completes drag-and-drop 1 step 433ms
🟢 act completes kanban-board 1 step 587ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 448ms
🟢 act completes otp-auto-advance 1 step 690ms
🟢 act completes date-picker 1 step 811ms
🟢 act completes rich-text-editor 1 step 10.1s
🟢 act completes playbook-cleanup 1 step 566ms
🟢 act completes promo-storefront 1 step 2.3s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 223ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 290ms
🟢 act adds an address with a run-unique label 1 step 290ms

e2e 0.22.0 · 2m 2s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 21 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .changeset/last-read-grace.md Resolved
Comment thread .changeset/mobile-reads-budget.md Resolved
Comment thread packages/e2e/src/expect/async.ts Resolved
Comment thread docs/mobile.mdx Resolved
Comment thread packages/e2e/src/internal/time.ts Resolved
Comment thread packages/e2e/src/locator/engine.ts Resolved
Comment thread .changeset/last-read-grace.md Resolved
…to deadline

Also keep one read window across stale retries and name locator.waitFor in docs.
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 36 model calls · 191.6k tokens (77% cached) · $0.0151

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 36 calls 4m 37s
🟢 act completes Login Form 1 step 19.2s
🟢 act completes Infinite Scroll List 1 step 20.1s
🟢 act completes Modal Flow 1 step 10.6s
🟢 act completes Bottom Tabs 1 step 6.3s
🟢 act completes Text Input Variations 1 step 5.7s
🟢 act completes Broken Accessibility 2 steps · 6 calls 17.6s
🟢 act completes Vision Only 2 steps · 1 call 6.6s
🟢 act completes Gestures 1 step · 9 calls 23.6s
🟢 act completes Async States 1 step · 9 calls 19.6s
🟢 act completes Debounced Search 1 step 6.4s
🟢 act completes Huge Virtualized List 1 step 57.2s
🟢 act completes Flattened Registration Form 2 steps · 1 call 7.2s
🟢 act completes Flattened Login 2 steps · 6 calls 16.1s
🟢 act completes Sticky Chrome Target 1 step 6.9s
🟢 act completes WebView Accessibility 1 step · 4 calls 14.3s
🟢 act completes Permission Prompt 1 step 6.2s
🟢 act completes Product Catalog 1 step 8.0s
🟢 act completes Error Recovery 1 step 8.7s
🟢 act completes Choice Controls 1 step 7.9s
🟢 act completes Sequential Onboarding 1 step 9.1s

e2e 0.22.0 · 4m 38s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 2s
🟢 control inventory › a label query resolves a labeled text field 11.1s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 16.0s
🟢 control inventory › a placeholder query and attribute reads find the field 4.8s
🟢 control inventory › a secure field takes a fill and denies value reads 8.4s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.9s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.4s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.9s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.4s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 13.0s
🟢 control inventory › a radio group reports the checked option 9.7s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.7s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.9s
🟢 control inventory › a header answers its accessible name 4.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.2s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.5s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.7s
🟢 control inventory › pointer verbs land at the point asked for 10.0s
🟢 control inventory › an ambiguous locator fails at once 6.1s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 11.8s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 424ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 34s
🟢 device fixture › setAppearance flips the color scheme the app reads 14.7s
🟢 device fixture › setOrientation rotates the window the app measures 8.9s
🟢 device fixture › the clipboard reads back what was written 6.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.3s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.2s
🟢 device fixture › closeApp leaves the next commands on the same device 12.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.6s
🟢 device fixture › device.back pops the scenario 5.5s
🟢 device fixture › unstable_raw reaches the test session through agent-device 4.6s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.9s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.5s
🟢 tests/device.e2e.ts · 3 passed 1m 9s
🟢 sequential onboarding echoes the exact values 42.6s
🟢 onboarding rejects a malformed email before advancing 14.7s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.6s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 8s
🟢 infinite scroll reaches the target item 23.8s
🟢 product catalog adds exactly two of the right variant 10.7s
🟢 gestures: long-press, then swipe left 9.1s
🟢 sticky chrome: accept below the fold, then continue 10.8s
🟢 async states: load, pull to refresh, claim through the toast 11.2s
🟢 huge virtualized list reaches row 512 1m 51s
🟢 web view coupon form applies the code shown on the page 11.6s
🟢 tests/flows.e2e.ts · 7 passed 1m 10s
🟢 text inputs unlock the submit button 13.7s
🟢 filled text inputs report their values 10.2s
🟢 modal flow confirms through the native alert 10.3s
🟢 bottom tabs act inside the last tab 7.5s
🟢 error recovery retries, then confirms the delete 9.6s
🟢 choice controls place the exact order 7.5s
🟢 debounced search selects the target once results arrive 11.3s
🟢 tests/login-form.e2e.ts · 3 passed 46.8s
🟢 login form › rejects a malformed email 13.5s
🟢 login form › rejects an unknown account 13.7s
🟢 login form › signs in and logs out 19.6s
🟢 tests/smoke.e2e.ts · 1 passed 7.2s
🟢 home lists the scenarios and opens one 7.2s

e2e 0.22.0 · 5m 51s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 22s
🟢 control inventory › a label query resolves a labeled text field 17.6s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 9.7s
🟢 control inventory › a placeholder query and attribute reads find the field 4.0s
🟢 control inventory › a secure field takes a fill and denies value reads 5.6s
🟢 control inventory › press sends a character and Enter through the soft keyboard 6.7s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 6.6s
🟢 control inventory › toBeFocused follows the focus 7.2s
🟢 control inventory › a switch checks, unchecks, and reports its state 7.5s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.3s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 9.5s
🟢 control inventory › a tab role answers getByRole 3.1s
🟢 control inventory › a radio group reports the checked option 7.1s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.3s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 5.7s
🟢 control inventory › a header answers its accessible name 3.3s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 4.9s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 4.8s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.7s
🟢 control inventory › a double tap lands inside the 300 ms window 6.1s
🟢 control inventory › an ambiguous locator fails at once 4.5s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 5.9s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 9s
🟢 device fixture › setOrientation rotates the window the app measures 4.8s
🟢 device fixture › home leaves the app, openApp brings it back where it was 3.5s
🟢 device fixture › foregroundApp sees the home screen after home() 3.5s
🟢 device fixture › closeApp then openApp relaunches on the home list 5.4s
🟢 device fixture › closeApp leaves the next commands on the same device 5.7s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 3.8s
🟢 device fixture › device.back pops the scenario 3.9s
🟢 device fixture › unstable_raw reaches the test session through agent-device 3.1s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 14.9s
🟢 device fixture › setLocation and clearLocation change what the app reads 20.2s
🟢 tests/device.e2e.ts · 5 passed 35.1s
🟢 permission prompt › allowing the system dialog enables the microphone 5.0s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 10.5s
🟢 sequential onboarding echoes the exact values 9.1s
🟢 onboarding rejects a malformed email before advancing 4.9s
🟢 dismissing the keyboard over choice controls leaves them as they were 5.7s
🟢 tests/flows-lists.e2e.ts · 7 passed 1m 43s
🟢 infinite scroll reaches the target item 14.3s
🟢 product catalog adds exactly two of the right variant 7.8s
🟢 gestures: long-press, then swipe left 7.3s
🟢 gestures: double-tap completes the flow 9.0s
🟢 sticky chrome: accept below the fold, then continue 8.0s
🟢 async states: load, pull to refresh, claim through the toast 10.6s
🟢 huge virtualized list reaches row 512 46.3s
🟢 tests/flows.e2e.ts · 7 passed 45.4s
🟢 text inputs unlock the submit button 5.5s
🟢 filled text inputs report their values 4.9s
🟢 modal flow confirms through the native alert 7.7s
🟢 bottom tabs act inside the last tab 6.0s
🟢 error recovery retries, then confirms the delete 7.9s
🟢 choice controls place the exact order 5.4s
🟢 debounced search selects the target once results arrive 7.9s
🟢 tests/login-form.e2e.ts · 3 passed 24.8s
🟢 login form › rejects a malformed email 6.5s
🟢 login form › rejects an unknown account 6.6s
🟢 login form › signs in and logs out 11.7s
🟢 tests/smoke.e2e.ts · 1 passed 4.6s
🟢 home lists the scenarios and opens one 4.6s

e2e 0.22.0 · 7m 4s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 12 replayed from cache · 59 model calls · 522.0k tokens (80% cached) · $0.0380

⚠️ 1 flaky test passed on a retry

⚠️ act completes Flattened Login
tests-agent/scenarios.e2e.ts:266

AUTOMATION_UNSUPPORTED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: log in with the email and password the screen shows as a hint. The screen lists nothing useful; take a screenshot and work from it.", after 1m 24s and 14 model calls

agent.act blocked: The email hint was entered, but the iOS keyboard’s typing-predictions introduction overlay appeared and intercepts interaction with the password field and Continue control. Keyboard dismissal and tapping the overlay’s Continue control were prevented, so login could not be completed.

  • Turn 11: press({"key":"Enter"}) → [press] press Enter on the focused field failed: POLICY_DENIED: keyboard.press failed: Unable to press the iOS keyboard return key
  • Turn 12: press({"key":"Escape"}) → [press] press Escape on the focused field failed: POLICY_DENIED: agent-device cannot press Escape on a device surface; only Enter, Space, and single characters…
  • Turn 13: swipe({"from":{"x":300,"y":500},"to":{"x":300,"y":740}}) → [swipe] Tool "swipe" failed: swipe failed: Gesture trajectory does not fit inside the viewport Hint: Move the gesture away from the edge or reduce its translat…
  • Screen: dev.e2e.benchmark / Flattened Login

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 59 calls 8m 22s
🟢 act completes Login Form 1 step 37.0s
🟢 act completes Infinite Scroll List 1 step 26.0s
🟢 act completes Modal Flow 1 step 9.8s
🟢 act completes Bottom Tabs 1 step 7.3s
🟢 act completes Text Input Variations 1 step 13.3s
🟢 act completes Broken Accessibility 2 steps · 5 calls 24.9s
🟢 act completes Vision Only 2 steps · 6 calls 29.9s
🟢 act completes Gestures 1 step · 7 calls 31.0s
🟢 act completes Async States 1 step · 11 calls 47.0s
🟢 act completes Debounced Search 1 step · 1 call 27.4s
🟢 act completes Flattened Registration Form 2 steps · 1 call 28.3s
⚠️ act completes Flattened Login (1 failed attempt first) 2 steps · 11 calls 59.3s
🟢 act completes Sticky Chrome Target 1 step 10.3s
🟢 act completes WebView Accessibility 1 step 11.7s
🟢 act completes Permission Prompt 1 step 10.6s
🟢 act completes Photo Picker 1 step · 2 calls 29.5s
🟢 act completes Product Catalog 1 step 10.0s
🟢 act completes Error Recovery 1 step 9.4s
🟢 act completes Choice Controls 1 step · 7 calls 37.3s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 32.2s
🟢 act completes Apple Pay 1 step 10.3s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

e2e 0.22.0 · 9m 52s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/e2e/src/locator/engine.ts Resolved
@github-actions github-actions Bot added size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile and removed Ready for Human Review size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile labels Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound mobile reads by the operation budget without turning a slow last poll into a timeout

1 participant