Skip to content

fix(mobile): read Compose Multiplatform text fields on iOS - #1427

Merged
okwasniewski merged 1 commit into
mainfrom
fix/mobile-compose-text-fields
Oct 28, 2026
Merged

okwasniewski merged 1 commit into
mainfrom
fix/mobile-compose-text-fields

Conversation

@kacperkapusciak

@kacperkapusciak kacperkapusciak commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Fixes the iOS Kotlin Multiplatform job in last night's examples nightly (run 40592743298). Also fixes the iOS Kotlin Multiplatform and Flutter fill failures that #1410 brought to main.

What changed

  • Placeholder (packages/mobile/src/nodes.ts). On iOS, an empty Compose Multiplatform field reports no placeholder. It adds the text it draws to its label instead, as "Name, Ada Lovelace", over one StaticText child. The agent read that as a filled field and tried to clear it until the step timed out. Now, on an empty, editable iOS field, a trailing ", <text>" that matches the field's only text child becomes the placeholder attribute, and that child is left out. The agent sees textbox "Name" testid="name".
  • fill / clear guard (packages/mobile/src/surface.ts). fix(mobile): revalidate input bindings before fill and clear #1410 refused every role except textbox, searchbox and combobox. agent-device 0.21.33's simulator AX tree reports Compose Multiplatform and Flutter fields as an unclassified Other. So the guard now also lets through an iOS other that holds a value or only text, and checks again against the snapshot taken just before typing. A childless, valueless other (a React Native Pressable) and a container of controls are still refused.
  • The mobile reference docs, both copies of the skill's setup.md, and a changeset are updated.

The placeholder logic needs the field to read as editable. agent-device's XCTest tree reports it that way already. Its simulator AX tree, which the nightly uses, will only do so after callstack/agent-device#4008. A follow-up PR bumps agent-device once that ships.

Verified

Ran it locally: yes

  • examples/with-kotlin-multiplatform on an iPhone 18 Pro simulator (iOS 27). The suite ran with packages packed from main and from this branch, both with agent-device 0.21.33, using e2e run --target ios --no-cache --retries 0 --video.
main this branch
main branch

main: 2 failed, 3 passed.

UNSUPPORTED_CAPABILITY: agent-device can only fill editable fields; node n70 is other
AUTOMATION_UNSUPPORTED: agent.act blocked: ... the agent could not replace the existing “Ada Lovelace” value with “Grace” ...

This branch: 5 passed, including both agent tests with live model calls. The same suite also passes 5 of 5 against the agent-device build from #4008.

Not run locally: @e2e-dev/testbed, because another checkout was holding its port, and the mobile benchmark. The mobile benchmark runs in the mobile workflow on this PR.

greets-main.mp4
greets-branch.mp4

View guided diff Turn on auto-fix

An empty Compose Multiplatform field on iOS reports no placeholder; it
appends the text it draws to its label, so "Name, Ada Lovelace" read as a
field holding a name and the examples nightly's agent tried to clear it.
Read that trailing text as the field's placeholder when it is the field's
only child. It applies once the field reads editable: agent-device's XCTest
tree reports it so today, its simulator AX tree once
callstack/agent-device types text-entry-trait elements (follow-up bump).

#1410 refused fill and clear on anything but a text input role, which broke
the Compose Multiplatform and Flutter iOS examples: agent-device 0.21.33's
simulator AX tree reports their fields as an unclassified Other. Type into
an Other that holds nothing but text; a container of controls stays refused.
@github-actions github-actions Bot added the size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 41 model calls · 240.6k tokens (77% cached) · $0.0189

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 41 calls 6m 55s
🟢 act completes Login Form 1 step 30.2s
🟢 act completes Infinite Scroll List 1 step 25.2s
🟢 act completes Modal Flow 1 step 11.8s
🟢 act completes Bottom Tabs 1 step 8.6s
🟢 act completes Text Input Variations 1 step 9.9s
🟢 act completes Broken Accessibility 2 steps · 5 calls 22.1s
🟢 act completes Vision Only 2 steps · 1 call 7.4s
🟢 act completes Gestures 1 step · 7 calls 29.1s
🟢 act completes Async States 1 step · 16 calls 59.2s
🟢 act completes Debounced Search 1 step 8.0s
🟢 act completes Huge Virtualized List 1 step 1m 18s
🟢 act completes Flattened Registration Form 2 steps · 1 call 10.9s
🟢 act completes Flattened Login 2 steps · 6 calls 24.4s
🟢 act completes Sticky Chrome Target 1 step 9.2s
🟢 act completes WebView Accessibility 1 step · 5 calls 26.9s
🟢 act completes Permission Prompt 1 step 8.7s
🟢 act completes Product Catalog 1 step 10.4s
🟢 act completes Error Recovery 1 step 9.9s
🟢 act completes Choice Controls 1 step 10.5s
🟢 act completes Sequential Onboarding 1 step 14.6s

e2e 0.21.3 · 6m 58s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 8s
🟢 control inventory › a label query resolves a labeled text field 13.3s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 17.4s
🟢 control inventory › a placeholder query and attribute reads find the field 5.0s
🟢 control inventory › a secure field takes a fill and denies value reads 8.9s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.9s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.4s
🟢 control inventory › toBeFocused follows the focus 9.4s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.9s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.6s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 13.4s
🟢 control inventory › a radio group reports the checked option 10.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.0s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.3s
🟢 control inventory › a header answers its accessible name 4.2s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.1s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.2s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.3s
🟢 control inventory › pointer verbs land at the point asked for 9.8s
🟢 control inventory › an ambiguous locator fails at once 6.2s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 11.8s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 616ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 40s
🟢 device fixture › setAppearance flips the color scheme the app reads 17.5s
🟢 device fixture › setOrientation rotates the window the app measures 10.8s
🟢 device fixture › the clipboard reads back what was written 6.2s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.0s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.1s
🟢 device fixture › closeApp leaves the next commands on the same device 11.6s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.6s
🟢 device fixture › device.back pops the scenario 6.4s
🟢 device fixture › unstable_raw reaches the test session through agent-device 5.2s
🟢 device fixture › setLocation and clearLocation change what the app reads 11.3s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.7s
🟢 tests/device.e2e.ts · 3 passed 1m 8s
🟢 sequential onboarding echoes the exact values 42.1s
🟢 onboarding rejects a malformed email before advancing 14.2s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.4s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 24s
🟢 infinite scroll reaches the target item 26.1s
🟢 product catalog adds exactly two of the right variant 9.9s
🟢 gestures: long-press, then swipe left 9.0s
🟢 sticky chrome: accept below the fold, then continue 11.4s
🟢 async states: load, pull to refresh, claim through the toast 12.7s
🟢 huge virtualized list reaches row 512 2m 2s
🟢 web view coupon form applies the code shown on the page 13.0s
🟢 tests/flows.e2e.ts · 7 passed 1m 11s
🟢 text inputs unlock the submit button 13.0s
🟢 filled text inputs report their values 10.1s
🟢 modal flow confirms through the native alert 9.9s
🟢 bottom tabs act inside the last tab 8.0s
🟢 error recovery retries, then confirms the delete 10.2s
🟢 choice controls place the exact order 7.8s
🟢 debounced search selects the target once results arrive 11.6s
🟢 tests/login-form.e2e.ts · 3 passed 51.0s
🟢 login form › rejects a malformed email 13.9s
🟢 login form › rejects an unknown account 14.1s
🟢 login form › signs in and logs out 22.9s
🟢 tests/smoke.e2e.ts · 1 passed 7.7s
🟢 home lists the scenarios and opens one 7.7s

e2e 0.21.3 · 6m 12s · ios-simulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 54s
🟢 control inventory › a label query resolves a labeled text field 24.9s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 11.4s
🟢 control inventory › a placeholder query and attribute reads find the field 7.7s
🟢 control inventory › a secure field takes a fill and denies value reads 8.1s
🟢 control inventory › press sends a character and Enter through the soft keyboard 8.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 8.3s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 8.3s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.5s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.3s
🟢 control inventory › a tab role answers getByRole 3.8s
🟢 control inventory › a radio group reports the checked option 7.6s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.8s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.5s
🟢 control inventory › a header answers its accessible name 4.2s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.4s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.5s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.4s
🟢 control inventory › a double tap lands inside the 300 ms window 6.5s
🟢 control inventory › an ambiguous locator fails at once 5.0s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.3s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 19s
🟢 device fixture › setOrientation rotates the window the app measures 5.1s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.8s
🟢 device fixture › foregroundApp sees the home screen after home() 4.0s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.9s
🟢 device fixture › closeApp leaves the next commands on the same device 7.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.2s
🟢 device fixture › device.back pops the scenario 4.7s
🟢 device fixture › unstable_raw reaches the test session through agent-device 3.8s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.0s
🟢 device fixture › setLocation and clearLocation change what the app reads 23.1s
🟢 tests/device.e2e.ts · 5 passed 42.0s
🟢 permission prompt › allowing the system dialog enables the microphone 6.5s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 12.9s
🟢 sequential onboarding echoes the exact values 10.0s
🟢 onboarding rejects a malformed email before advancing 6.0s
🟢 dismissing the keyboard over choice controls leaves them as they were 6.6s
🟢 tests/flows-lists.e2e.ts · 7 passed 2m 4s
🟢 infinite scroll reaches the target item 17.2s
🟢 product catalog adds exactly two of the right variant 8.5s
🟢 gestures: long-press, then swipe left 8.1s
🟢 gestures: double-tap completes the flow 9.6s
🟢 sticky chrome: accept below the fold, then continue 9.0s
🟢 async states: load, pull to refresh, claim through the toast 11.8s
🟢 huge virtualized list reaches row 512 59.6s
🟢 tests/flows.e2e.ts · 7 passed 51.9s
🟢 text inputs unlock the submit button 6.3s
🟢 filled text inputs report their values 6.1s
🟢 modal flow confirms through the native alert 9.5s
🟢 bottom tabs act inside the last tab 6.7s
🟢 error recovery retries, then confirms the delete 8.8s
🟢 choice controls place the exact order 5.6s
🟢 debounced search selects the target once results arrive 8.9s
🟢 tests/login-form.e2e.ts · 3 passed 27.1s
🟢 login form › rejects a malformed email 7.4s
🟢 login form › rejects an unknown account 7.1s
🟢 login form › signs in and logs out 12.6s
🟢 tests/smoke.e2e.ts · 1 passed 6.0s
🟢 home lists the scenarios and opens one 6.0s

e2e 0.21.3 · 8m 24s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 6 replayed from cache · 74 model calls · 450.5k tokens (84% cached) · $0.0285

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 74 calls 11m 58s
🟢 act completes Login Form 1 step 43.0s
🟢 act completes Infinite Scroll List 1 step · 1 call 44.8s
🟢 act completes Modal Flow 1 step · 1 call 26.5s
🟢 act completes Bottom Tabs 1 step · 1 call 24.0s
🟢 act completes Text Input Variations 1 step · 4 calls 41.6s
🟢 act completes Broken Accessibility 2 steps · 6 calls 30.5s
🟢 act completes Vision Only 2 steps · 6 calls 33.0s
🟢 act completes Gestures 1 step · 12 calls 1m 7s
🟢 act completes Async States 1 step · 7 calls 29.4s
🟢 act completes Debounced Search 1 step · 1 call 28.5s
🟢 act completes Flattened Registration Form 2 steps · 1 call 30.6s
🟢 act completes Flattened Login 2 steps · 10 calls 1m 2s
🟢 act completes Sticky Chrome Target 1 step · 2 calls 56.7s
🟢 act completes WebView Accessibility 1 step 13.8s
🟢 act completes Permission Prompt 1 step 11.9s
🟢 act completes Photo Picker 1 step · 2 calls 32.5s
🟢 act completes Product Catalog 1 step 10.5s
🟢 act completes Error Recovery 1 step 10.0s
🟢 act completes Choice Controls 1 step · 6 calls 40.7s
🟢 act completes Stripe PaymentSheet 1 step · 13 calls 47.5s
🟢 act completes Apple Pay 1 step · 1 call 33.7s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

e2e 0.21.3 · 11m 59s · ios-simulator · run artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants