Skip to content

perf(agent): wait 1 s, not 2 s, after a caret key in a focused text field - #1630

Open
kdy1 wants to merge 2 commits into
mainfrom
perf/agent-caret-change-wait
Open

kdy1 wants to merge 2 commits into
mainfrom
perf/agent-caret-change-wait

Conversation

@kdy1

@kdy1 kdy1 commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator
  • After press or pressKey, a settled look waits up to 2 s (FULL_CHANGE_WAIT_MS) for the tree to change shape. A caret key (ArrowLeft, ArrowRight, Home, End, Control/Meta+a, with Shift or Control) in a textbox or searchbox that already holds the focus usually changes nothing the tree records, so the whole window was spent.
  • That wait is now 1 s (HELD_STILL_MS) for those keys: changeWaitAfter in packages/core/src/agent/settle-policy.ts. The first version used 150 ms; review showed a selectionchange handler that reveals a toolbar 700 ms later was missed, so the window now covers it.
  • Kept at 2 s: taps, the first key or tap that gives a field the focus, combobox, Alt chords, Tab and the vertical keys, and every other action.
  • armChange (packages/core/src/agent/observation-feed.ts) keeps the later deadline when a window is still pending. Without it, tap(Save) then pressKey('End') in one turn would cut the tap's 2 s window. focusedNode() returns a node only when exactly one node has the focus.
  • Known limit: a handler that updates the tree more than 1 s after the key is read on the next look, as after any action with a window shorter than its delay.

Verified

Ran it locally: yes

  • Surface: the web benchmark's tests-agent/ replay with --strict-cache, through pnpm bench:ab --config e2e.agent.config.ts against the merge base (e27f492c), 6 pairs in random order.
  • The 7 tests that call the model on a replay (3 planted-bug judgments, 4 act completes … scenarios) are left out of bench:ab with --grep-invert, since their model calls differ between runs. They ran once on the 150 ms version of this branch with a key (CI=1 e2e run --config e2e.agent.config.ts --ai-trace, recordings read-only): 33 tests passed, no REPLAY_STALE. On this head CI's web agent job ran them with a key: 33 passed, 20 model calls. The --no-cache path was not run.
  • Behavior: every test ran the same steps with the same outcome (status, cache mode, replayed actions) on every run of both builds.
Metric main this branch change (95% CI)
wall (process) 68.71 s 62.86 s -9.0% (-11.4% to -5.8%)
suite (sum of attempts) 67.52 s 61.67 s -9.2% (-11.6% to -5.9%)
observe (events) 38.70 s 32.78 s -15.3% (-15.6% to -15.0%)
action (events) 25.89 s 25.64 s same (-7.8% to +6.4%)
act completes rich-text-editor 20.49 s 14.58 s -28.9% (-29.3% to -28.4%)
  • Review regression test: tests/integration/agent-caret-settle.test.ts has a fake-engine popup that opens 700 ms after press and after pressKey on a focused field. Both cases fail at the 150 ms window and pass at 1 s. The same file keeps a first tap, a tap on the focused field, a caret key before the field held the focus, and a batched tap then key at the full window.
  • After merging main (the agent moved into @e2e-dev/core, refactor: move e2e's core into @e2e-dev/core, and engines peer on it #1737): e2e unit 3707 pass, agent integration 228 pass, pnpm check exit 0. The timings above were measured before that move, on 1112e50a against e27f492c; they were not re-measured on the merged tree. The iOS and Android agent and deterministic suites passed in CI on an earlier head, since settle-policy.ts is shared with mobile.
  • The replay suite cannot catch a delayed effect: every changing look in it ends within about 120 ms, which is why the fake-engine file exists.

At 150 ms the replay took 59.75 s (-16.1%) and the rich-text step 9.40 s. Taps stay at 2 s for the same reason: a click can open a panel that renders late. Five dead looks of about 2.1 s (10.7 s) remain: right clicks in the context-menu trap, the upload tap, the gift-card amount tap, and the editor's bold tap.

🤖 Generated with Claude Code

@kdy1
kdy1 requested a review from okwasniewski as a code owner November 8, 2026 07:46
@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 93 passed

All 93 tests in 42 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 568ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 432ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 136ms
🟢 tests/async-states.e2e.ts · 2 passed 7.1s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.3s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 251ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 136ms
🟢 canvas only › a shape out of order paints the error and resets progress 115ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 373ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 220ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 153ms
🟢 tests/cart-totals.e2e.ts · 2 passed 373ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 203ms
🟢 cart totals › a quantity never drops below one 170ms
🟢 tests/checkout-review.e2e.ts · 1 passed 145ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 145ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 495ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 268ms
🟢 context menu trap › the decoy menu items refuse and keep the file 227ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.5s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 235ms
🟢 control inventory › check picks a radio the pick replaces with its summary 137ms
🟢 control inventory › a long press and a double click are told apart from a tap 977ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 273ms
🟢 control inventory › setInputFiles attaches files named from the project root 152ms
🟢 control inventory › the details view is left through the browser history 160ms
🟢 control inventory › scrollIntoView brings the footnote into view 252ms
🟢 control inventory › every exercise done shows the success message 1.3s
🟢 tests/css-content-ui.e2e.ts · 2 passed 348ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 194ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 154ms
🟢 tests/date-picker.e2e.ts · 2 passed 708ms
🟢 date picker › navigates three months ahead and books the required date 427ms
🟢 date picker › confirming without the exact date shows the error 281ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 678ms
🟢 div soup › a wrong code is rejected 308ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 370ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 642ms
🟢 drag and drop › an out-of-order item resets the dropzone 314ms
🟢 drag and drop › dropping Banana then Cherry passes 328ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 228ms
🟢 filter deep link › filtering writes the query to the URL 131ms
🟢 filter deep link › opening a shared link restores the filter 97ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 503ms
🟢 gift card purchase › refuses to buy without an amount 138ms
🟢 gift card purchase › refuses a recipient without an at sign 181ms
🟢 gift card purchase › sends the gift card 184ms
🟢 tests/hover-menu.e2e.ts · 2 passed 409ms
🟢 hover menu › a decoy leaf is refused 177ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 232ms
🟢 tests/iframe-form.e2e.ts · 2 passed 460ms
🟢 iframe form › a wrong code is refused by the host page 213ms
🟢 iframe form › the code read from one frame applies through the other 247ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 421ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 98ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 158ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 165ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.6s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.6s
🟢 tests/kanban-board.e2e.ts · 3 passed 669ms
🟢 kanban board › pointer drags move both cards into the goal layout 352ms
🟢 kanban board › a swipe along a path drags a card as well 134ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 183ms
🟢 tests/login-form.e2e.ts · 4 passed 714ms
🟢 login form › rejects a malformed email 163ms
🟢 login form › rejects a short password 169ms
🟢 login form › rejects an unknown account 185ms
🟢 login form › signs in and logs out 197ms
🟢 tests/lying-labels.e2e.ts · 4 passed 650ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 132ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 102ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 217ms
🟢 lying labels › visible text drives the payment through 199ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 446ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 152ms
🟢 native dialogs › dismissing the confirm cancels the order 144ms
🟢 native dialogs › a wrong gift code is rejected 150ms
🟢 tests/new-tab-flow.e2e.ts · 2 passed 428ms
🟢 new tab flow › reads the code in the tab the app opens and enters it back in the original tab 214ms
🟢 new tab flow › a tap that opens a tab leaves the screen on the original tab, and activate shows the new one 214ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.3s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.3s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.3s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 265ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 312ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.3s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.3s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 177ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 472ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 142ms
🟢 otp auto-advance › a wrong code clears every box and starts over 764ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 642ms
🟢 continues only after the banner and the overlay are gone 642ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 429ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 210ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 219ms
🟢 tests/price-sorting.e2e.ts · 3 passed 361ms
🟢 price sorting › opens in the featured order with sale prices struck through 105ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 122ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 134ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.3s
🟢 buys a deal once the promo chrome is out of the way 1.3s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 428ms
🟢 rich text editor › bolding exactly the required word passes the check 214ms
🟢 rich text editor › bold that spills into the previous word fails the check 214ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 516ms
🟢 saved addresses › adds an address and clears the form 173ms
🟢 saved addresses › ignores an add with an empty field 163ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 180ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 516ms
🟢 shadow DOM form › rejects a wrong access code 249ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 267ms
🟢 tests/smoke.e2e.ts · 1 passed 394ms
🟢 home lists the scenarios and opens one 394ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.1s
🟢 taps the moving target three times across rebuilds of the list 2.1s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 205ms
🟢 accepts the terms once the button clears the sticky footer 205ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 14.6s
🟢 scrolls the windowed table to the Golden Row and claims it 14.6s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 262ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.2s

Tested by e2e 0.23.1 · 1m 1s · web · run artifacts

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 25 model calls · 117.4k tokens (87% cached) · $0.0065

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.6s
🟢 control inventory › check sets the checkbox and the radio 1 step 669ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 257ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 874ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.5s
🟢 control inventory › upload attaches the named project files 1 step 2.5s
🟢 control inventory › back returns from the details view to the inventory 1 step 404ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 410ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 455ms
🟢 act signs in with the benchmark account 1 step 455ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 14.3s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 7.8s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.6s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 4.9s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 17 calls 1m 16s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.5s
🟢 act completes lying-labels 1 step · 3 calls 4.8s
🟢 act completes iframe-form 1 step · 3 calls 4.7s
🟢 act completes infinite-scroll 1 step 3.4s
🟢 act completes virtualized-table 1 step 18.8s
🟢 act completes overlay-trap 1 step 900ms
🟢 act completes sticky-chrome 1 step 920ms
🟢 act completes async-states 1 step · 3 calls 6.8s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 481ms
🟢 act completes drag-and-drop 1 step 515ms
🟢 act completes kanban-board 1 step 739ms
🟢 act completes context-menu-trap 1 step 2.7s
🟢 act completes stale-dom 1 step 421ms
🟢 act completes otp-auto-advance 1 step 685ms
🟢 act completes date-picker 1 step 776ms
🟢 act completes rich-text-editor 1 step 8.0s
🟢 act completes playbook-cleanup 1 step 526ms
🟢 act completes promo-storefront 1 step 2.3s
🟢 act completes new-tab-flow 1 step · 5 calls 9.7s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 205ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 262ms
🟢 act adds an address with a run-unique label 1 step 262ms

Tested by e2e 0.23.1 · 1m 39s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .changeset/caret-change-wait.md Resolved
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 59 model calls · 385.2k tokens (83% cached) · $0.0249

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 59 calls 8m 32s
🟢 act completes Login Form 1 step 39.7s
🟢 act completes Infinite Scroll List 1 step 39.0s
🟢 act completes Modal Flow 1 step 10.5s
🟢 act completes Bottom Tabs 1 step 7.9s
🟢 act completes Text Input Variations 1 step 13.2s
🟢 act completes Broken Accessibility 2 steps · 6 calls 25.9s
🟢 act completes Vision Only 2 steps · 6 calls 26.8s
🟢 act completes Gestures 1 step · 7 calls 30.1s
🟢 act completes Async States 1 step · 10 calls 39.4s
🟢 act completes Debounced Search 1 step · 1 call 27.3s
🟢 act completes Flattened Registration Form 2 steps · 1 call 25.5s
🟢 act completes Flattened Login 2 steps · 11 calls 55.7s
🟢 act completes Sticky Chrome Target 1 step 13.1s
🟢 act completes WebView Accessibility 1 step 12.4s
🟢 act completes Permission Prompt 1 step 11.8s
🟢 act completes Photo Picker 1 step · 2 calls 30.1s
🟢 act completes Product Catalog 1 step 10.2s
🟢 act completes Error Recovery 1 step 9.5s
🟢 act completes Choice Controls 1 step · 6 calls 35.9s
🟢 act completes Stripe PaymentSheet 1 step · 9 calls 33.3s
🟢 act completes Apple Pay 1 step 14.3s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.1 · 8m 33s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 33 model calls · 171.8k tokens (77% cached) · $0.0135

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 33 calls 5m 26s
🟢 act completes Login Form 1 step 22.3s
🟢 act completes Infinite Scroll List 1 step 21.5s
🟢 act completes Modal Flow 1 step 9.7s
🟢 act completes Bottom Tabs 1 step 6.6s
🟢 act completes Text Input Variations 1 step 5.7s
🟢 act completes Broken Accessibility 2 steps · 5 calls 13.8s
🟢 act completes Vision Only 2 steps · 1 call 6.1s
🟢 act completes Gestures 1 step · 7 calls 18.7s
🟢 act completes Async States 1 step · 8 calls 17.4s
🟢 act completes Debounced Search 1 step 6.6s
🟢 act completes Huge Virtualized List 1 step 1m 49s
🟢 act completes Flattened Registration Form 2 steps · 1 call 8.3s
🟢 act completes Flattened Login 2 steps · 6 calls 16.5s
🟢 act completes Sticky Chrome Target 1 step 8.8s
🟢 act completes WebView Accessibility 1 step · 5 calls 14.5s
🟢 act completes Permission Prompt 1 step 6.1s
🟢 act completes Product Catalog 1 step 8.0s
🟢 act completes Error Recovery 1 step 8.8s
🟢 act completes Choice Controls 1 step 8.4s
🟢 act completes Sequential Onboarding 1 step 9.7s

Tested by e2e 0.23.1 · 5m 27s · android-emulator · run artifacts

@kdy1

kdy1 commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

[claude-sonnet-5-5] responding on behalf of kdy1

The red web agent job is not from this diff. act completes infinite-scroll fails with REPLAY_STALE (action-failed, scrollUntil stops after 13 screens) on every run of this PR, and it failed the same way on main at d784448 (run 42369507311). That step is the one #1614 fixes, and #1614's own web agent job is green. This diff only shortens the wait after a caret key in a focused text field, which that step never presses. The agent job is skipped on most runs, so main had no other run of it since the Playwright 1.65 bump.

The long-press step of control-inventory also failed once (the 563 ms hold registered as a tap) and passed on rerun.

Left for a human: merge #1614, then this PR needs a rebase and one rerun of web agent before it can be labeled.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 58 passed, 1 skipped

All 59 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 31s
🟢 control inventory › a label query resolves a labeled text field 21.8s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 11.0s
🟢 control inventory › a placeholder query and attribute reads find the field 5.6s
🟢 control inventory › a secure field takes a fill and denies value reads 5.7s
🟢 control inventory › press sends a character and Enter through the soft keyboard 7.1s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 7.3s
🟢 control inventory › toBeFocused follows the focus 7.3s
🟢 control inventory › a switch checks, unchecks, and reports its state 7.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.3s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 9.8s
🟢 control inventory › a tab role answers getByRole 3.5s
🟢 control inventory › a radio group reports the checked option 7.3s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.5s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 5.9s
🟢 control inventory › a header answers its accessible name 3.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 4.7s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 4.8s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.8s
🟢 control inventory › a double tap lands inside the 300 ms window 6.1s
🟢 control inventory › an ambiguous locator fails at once 4.5s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 6.1s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.3s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.0s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 13s
🟢 device fixture › setOrientation rotates the window the app measures 4.7s
🟢 device fixture › home leaves the app, openApp brings it back where it was 3.8s
🟢 device fixture › foregroundApp sees the home screen after home() 3.3s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.0s
🟢 device fixture › closeApp leaves the next commands on the same device 5.6s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.0s
🟢 device fixture › openLink opens the scenario a deep link names 4.1s
🟢 device fixture › device.back pops the scenario 3.9s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 3.4s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 14.3s
🟢 device fixture › setLocation and clearLocation change what the app reads 20.3s
🟢 tests/device.e2e.ts · 5 passed 36.6s
🟢 permission prompt › allowing the system dialog enables the microphone 5.2s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 10.9s
🟢 sequential onboarding echoes the exact values 9.1s
🟢 onboarding rejects a malformed email before advancing 5.3s
🟢 dismissing the keyboard over choice controls leaves them as they were 6.0s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 1s
🟢 infinite scroll reaches the target item 27.2s
🟢 product catalog adds exactly two of the right variant 8.1s
🟢 gestures: long-press, then swipe left 7.0s
🟢 gestures: double-tap completes the flow 9.1s
🟢 sticky chrome: accept below the fold, then continue 10.3s
🟢 async states: load, pull to refresh, claim through the toast 11.2s
🟢 huge virtualized list reaches row 512 1m 48s
🟢 tests/flows.e2e.ts · 7 passed 46.8s
🟢 text inputs unlock the submit button 5.8s
🟢 filled text inputs report their values 5.1s
🟢 modal flow confirms through the native alert 8.0s
🟢 bottom tabs act inside the last tab 6.2s
🟢 error recovery retries, then confirms the delete 8.0s
🟢 choice controls place the exact order 5.7s
🟢 debounced search selects the target once results arrive 8.1s
🟢 tests/login-form.e2e.ts · 3 passed 25.7s
🟢 login form › rejects a malformed email 6.8s
🟢 login form › rejects an unknown account 6.7s
🟢 login form › signs in and logs out 12.2s
🟢 tests/smoke.e2e.ts · 1 passed 5.1s
🟢 home lists the scenarios and opens one 5.1s

Tested by e2e 0.23.1 · 8m 41s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 56 passed, 1 skipped

All 57 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 2m 58s
🟢 control inventory › a label query resolves a labeled text field 12.1s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.6s
🟢 control inventory › a placeholder query and attribute reads find the field 4.9s
🟢 control inventory › a secure field takes a fill and denies value reads 7.5s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.1s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 12.8s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.1s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 9.1s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.5s
🟢 control inventory › a radio group reports the checked option 9.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.2s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.0s
🟢 control inventory › a header answers its accessible name 4.1s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.1s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.3s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.1s
🟢 control inventory › pointer verbs land at the point asked for 9.0s
🟢 control inventory › an ambiguous locator fails at once 5.4s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.5s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 516ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.2s
🟢 tests/device-fixture.e2e.ts · 12 passed 1m 52s
🟢 device fixture › setAppearance flips the color scheme the app reads 14.9s
🟢 device fixture › setOrientation rotates the window the app measures 7.6s
🟢 device fixture › the clipboard reads back what was written 5.9s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.7s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.6s
🟢 device fixture › closeApp leaves the next commands on the same device 15.9s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.8s
🟢 device fixture › openLink opens the scenario a deep link names 14.0s
🟢 device fixture › device.back pops the scenario 7.4s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.3s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.1s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 9.8s
🟢 tests/device.e2e.ts · 3 passed 47.0s
🟢 sequential onboarding echoes the exact values 26.5s
🟢 onboarding rejects a malformed email before advancing 9.6s
🟢 dismissing the keyboard over choice controls leaves them as they were 10.9s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 0s
🟢 infinite scroll reaches the target item 33.9s
🟢 product catalog adds exactly two of the right variant 10.0s
🟢 gestures: long-press, then swipe left 8.7s
🟢 sticky chrome: accept below the fold, then continue 12.4s
🟢 async states: load, pull to refresh, claim through the toast 11.4s
🟢 huge virtualized list reaches row 512 2m 32s
🟢 web view coupon form applies the code shown on the page 12.0s
🟢 tests/flows.e2e.ts · 7 passed 1m 7s
🟢 text inputs unlock the submit button 12.0s
🟢 filled text inputs report their values 10.3s
🟢 modal flow confirms through the native alert 9.7s
🟢 bottom tabs act inside the last tab 7.4s
🟢 error recovery retries, then confirms the delete 9.2s
🟢 choice controls place the exact order 7.4s
🟢 debounced search selects the target once results arrive 11.5s
🟢 tests/login-form.e2e.ts · 3 passed 44.5s
🟢 login form › rejects a malformed email 12.7s
🟢 login form › rejects an unknown account 12.6s
🟢 login form › signs in and logs out 19.2s
🟢 tests/smoke.e2e.ts · 1 passed 8.2s
🟢 home lists the scenarios and opens one 8.2s

Tested by e2e 0.23.1 · 6m 40s · ios-simulator · run artifacts

@kdy1
kdy1 force-pushed the perf/agent-caret-change-wait branch from 34c5e74 to 23dc7e1 Compare November 9, 2026 02:49

@KrzysztofMoch KrzysztofMoch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 gpt-6.1-sol responding on behalf of KrzysztofMoch

Please add a regression test for delayed UI updates after a caret key in an already focused field, and adjust the settle policy so that observation captures the update before merging.

movesCaretOnly() treats the key and field role as enough evidence to shorten the wait, but moving a caret can trigger an asynchronous selectionchange handler. I reproduced this through the built CLI in real Chromium with a focused contenteditable textbox whose native selectionchange handler reveals a toolbar after 700 ms: this head returns the old tree after roughly 270 ms, while main captures the toolbar.

Both pressKey('Home') and press(target, 'Home') miss the update in all three repeats on this head and pass all three on main. The 50 ms control passes on both builds, giving 9/9 passes on main versus 3/9 on this head. This uses a scripted executor with no model calls, isolating the observation timing regression in a controlled browser fixture.

…ield

After press or pressKey, a settled look waits up to 2 s for the tree to
change shape. A caret key (ArrowLeft, ArrowRight, Home, End, select-all) in
a textbox or searchbox that already holds the focus usually changes nothing
the tree records, so the whole window was spent. It now waits 1 s
(HELD_STILL_MS), enough for a selectionchange handler that updates the tree
a beat later, such as a rich-text toolbar.

armChange keeps the later deadline of a window still pending, so a tap
followed by a caret key in one turn keeps the tap's full window.
focusedNode() names the focused node only when exactly one has the focus.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kdy1
kdy1 force-pushed the perf/agent-caret-change-wait branch from 23dc7e1 to 5fb3354 Compare November 10, 2026 05:38
@kdy1 kdy1 changed the title perf(agent): wait 150 ms, not 2 s, after a caret key in a focused text field perf(agent): wait 1 s, not 2 s, after a caret key in a focused text field Nov 10, 2026
@kdy1

kdy1 commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

[claude-sonnet-5-5] responding on behalf of kdy1

Fixed in 5fb3354. The caret window is now 1 s (HELD_STILL_MS), so a selectionchange handler that updates the tree 700 ms later is read. Regression test: agent-caret-settle.test.ts opens a popup 700 ms after press and after pressKey on a focused field; both fail at 150 ms and pass at 1 s. Replay is -9.0% wall against main (rich-text step 20.5 s to 14.6 s), 0 behavior differences. A handler slower than 1 s is still read late; that limit is in the PR body.

main moved the agent into @e2e-dev/core, so the two tests import it from
@e2e-dev/core/lib/agent/*, and the changeset names the package that now
holds settle-policy.ts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants