diff --git a/.changeset/ios-compose-text-fields.md b/.changeset/ios-compose-text-fields.md new file mode 100644 index 0000000000..f8ceb6454a --- /dev/null +++ b/.changeset/ios-compose-text-fields.md @@ -0,0 +1,5 @@ +--- +'@e2e-dev/mobile': patch +--- + +Read the placeholder an empty Compose Multiplatform field folds into its iOS label as the field's placeholder, so an empty field no longer reads as holding the example text. `fill` and `clear` type into an unclassified iOS element that holds a value or only text, the way a custom-drawn Compose Multiplatform or Flutter field can read. diff --git a/docs/reference/mobile.mdx b/docs/reference/mobile.mdx index 0d5fd73f91..163371fd3e 100644 --- a/docs/reference/mobile.mdx +++ b/docs/reference/mobile.mdx @@ -242,7 +242,9 @@ waits out the budget before the next action lands on it. `fill` and `clear` re-read the field before typing. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation. Non-input targets return -`UNSUPPORTED_CAPABILITY`. +`UNSUPPORTED_CAPABILITY`. An iOS `other`, the type of an element the platform +did not classify, is typed into when it holds a value or only text, since a +custom-drawn field (Compose Multiplatform, Flutter) can read that way. #### `videoTouches` @@ -355,7 +357,7 @@ running on the device. The next attempt waits for it to settle. | Contract member | On a device | | --- | --- | -| `observe` | The accessibility tree with rects. iOS: `Button` is `button`, `TextField` and `SearchField` are `textbox`, `SecureTextField` is a secure `textbox`, `Cell` is `listitem`, `StaticText` is `text`, `Switch` is `switch` with `checked`, `TabBar` and `SegmentedControl` are `tablist` and the buttons inside them `tab`, `ProgressIndicator` is `progressbar`, `ActivityIndicator` is `status`, `Stepper` is `spinbutton`, `Toolbar`, `Menu`, and `RadioGroup` are `toolbar`, `menu`, and `radiogroup`. Android: `TextView` is `text`, `EditText` is `textbox` and one UIAutomator flags as `password` is a secure `textbox` like `SecureTextField`, `Switch` and `CheckBox` keep their roles and `RadioButton` is `radio`, each with the `checked` state the device reports, layouts are `group`, `RecyclerView` and `AbsListView` (React Native's `FlatList`) are `list`, `TabLayout` and `BottomNavigationView` are `tablist` with their items as `tab`, `ProgressBar` is `progressbar`, `NumberPicker` is `spinbutton`, `Toolbar` and `RadioGroup` keep their names. Any other type is kept verbatim in lower case (`picker-wheel`), so it answers no vocabulary role, unless the platform reports it editable, which makes it a `textbox`. React Native on iOS has no trait for `checkbox`, `radio`, and a few other roles and spells the role and state into the accessibility value instead (`checkbox, unchecked`, `radio button, checked`); such a node takes that role, `checked` and `expanded` from the state words, and reports only what follows them as its value. An iOS field keeps a value equal to its label (an unlabeled field is named by its text); Android's echo of a text view's label as its value is dropped. The accessibility identifier (iOS) or resource id (Android) is the node's `testId`. An Android node is named by the content description its app set (React Native's `accessibilityLabel`), else by its text, and its `text` is what it shows, so a labeled text view or a filled field answers `getByLabel` by its label and `getByText` by its text; a view flagged as a heading is a `heading`, and the role description React Native writes for a role Android has no class for (`Tab`, `Tab List`, `Radio Group`) gives the node that role. A field's hint is its `placeholder` attribute, showing or not. An empty field has no value: Android flags a showing hint, and XCTest reports an empty iOS field's placeholder as its value, so an iOS field whose value equals its placeholder (trimmed) reads empty, text typed to match the placeholder exactly included. One Android fact is agent-device's, not the platform's: `rect` and every point are the screen's physical pixels; the display density it reports is not applied yet. Android's status bar and navigation bar, the full-width `com.android.systemui` windows along the screen's edges and the narrow status icons (clock, signal, battery) agent-device reports on their own inside that edge band, are left out; the notification shade, system dialogs, and a heads-up notification or other popup inset from the sides stay. Pixels on request, with every secure node's bounds painted black. | +| `observe` | The accessibility tree with rects. iOS: `Button` is `button`, `TextField` and `SearchField` are `textbox`, `SecureTextField` is a secure `textbox`, `Cell` is `listitem`, `StaticText` is `text`, `Switch` is `switch` with `checked`, `TabBar` and `SegmentedControl` are `tablist` and the buttons inside them `tab`, `ProgressIndicator` is `progressbar`, `ActivityIndicator` is `status`, `Stepper` is `spinbutton`, `Toolbar`, `Menu`, and `RadioGroup` are `toolbar`, `menu`, and `radiogroup`. Android: `TextView` is `text`, `EditText` is `textbox` and one UIAutomator flags as `password` is a secure `textbox` like `SecureTextField`, `Switch` and `CheckBox` keep their roles and `RadioButton` is `radio`, each with the `checked` state the device reports, layouts are `group`, `RecyclerView` and `AbsListView` (React Native's `FlatList`) are `list`, `TabLayout` and `BottomNavigationView` are `tablist` with their items as `tab`, `ProgressBar` is `progressbar`, `NumberPicker` is `spinbutton`, `Toolbar` and `RadioGroup` keep their names. Any other type is kept verbatim in lower case (`picker-wheel`), so it answers no vocabulary role, unless the platform reports it editable, which makes it a `textbox`. React Native on iOS has no trait for `checkbox`, `radio`, and a few other roles and spells the role and state into the accessibility value instead (`checkbox, unchecked`, `radio button, checked`); such a node takes that role, `checked` and `expanded` from the state words, and reports only what follows them as its value. An iOS field keeps a value equal to its label (an unlabeled field is named by its text); Android's echo of a text view's label as its value is dropped. The accessibility identifier (iOS) or resource id (Android) is the node's `testId`. An Android node is named by the content description its app set (React Native's `accessibilityLabel`), else by its text, and its `text` is what it shows, so a labeled text view or a filled field answers `getByLabel` by its label and `getByText` by its text; a view flagged as a heading is a `heading`, and the role description React Native writes for a role Android has no class for (`Tab`, `Tab List`, `Radio Group`) gives the node that role. A field's hint is its `placeholder` attribute, showing or not. Compose Multiplatform reports no placeholder on iOS: an empty field appends the text it draws to its label (`Name, Ada Lovelace`) over one `StaticText` child, so that text is the field's `placeholder` and the child is left out. An empty field has no value: Android flags a showing hint, and XCTest reports an empty iOS field's placeholder as its value, so an iOS field whose value equals its placeholder (trimmed) reads empty, text typed to match the placeholder exactly included. One Android fact is agent-device's, not the platform's: `rect` and every point are the screen's physical pixels; the display density it reports is not applied yet. Android's status bar and navigation bar, the full-width `com.android.systemui` windows along the screen's edges and the narrow status icons (clock, signal, battery) agent-device reports on their own inside that edge band, are left out; the notification shade, system dialogs, and a heads-up notification or other popup inset from the sides stay. Pixels on request, with every secure node's bounds painted black. | | `perform` | `tap`, `doubleTap`, `longPress`, `fill`, `clear`, `hover`, `check`, `uncheck`, `press` (`Enter`, `Space`, or one character), `swipe`, `dragTo`, and `focus` on editable fields. A tap on a `switch` or `checkbox` lands on its innermost control, since UIKit reports a settings row as one labelled switch with the real toggle as a child. `check` and `uncheck` need a `checked` state, which Android switches lack and a React Native checkbox or radio on iOS has. `doubleTap` is two presses, 284 to 285 ms apart on iOS (the cadence of agent-device's XCTest tap): agent-device's own double-tap gesture reaches a React Native `Pressable` as one press. A detector with a window of 300 ms or less can read the pair as two single taps. `longPress` holds for one second unless the test passes `duration`, past a React Native `Pressable`'s `delayLongPress` of 500 ms and the longer thresholds apps set; agent-device's own default hold is shorter and registers as a tap on those. `selectOption`, `setInputFiles`, `scrollIntoView`, and `secondaryTap` are not declared, and neither is `tapModifiers`, so `tap` or `doubleTap` with `modifiers` is `UNSUPPORTED_CAPABILITY`. | | `performAt` | `tap`, `doubleTap`, `longPress`, and `swipeTo` at a bare screen point, so `screen.tapAt`, `tap({ position })`, `screen.swipe({ from, to })`, and the agent's `tap_at` reach a control the tree does not list. No `hover`, `secondaryTap`, or `dragTo` at a point: touch has none without a node, so `hover_at` needs a listed control. | | `keyboard` | `type` and `press` into the focused field, the agent's `type` and `press` without a target and the fallback of `type_at` and `press_at`, and `dismiss`, the agent's `dismiss_keyboard`. | diff --git a/packages/mobile/src/nodes.ts b/packages/mobile/src/nodes.ts index 7ae8b2b08f..3022e8c6a9 100644 --- a/packages/mobile/src/nodes.ts +++ b/packages/mobile/src/nodes.ts @@ -94,8 +94,26 @@ const ROLE_MAP: Readonly> = { }; const SECURE_KINDS = new Set(['secure-text-field', 'securetextfield', 'password-field']); -/** Semantic roles that accept text input. */ -export const TEXT_INPUT_ROLES: ReadonlySet = new Set(['textbox', 'searchbox', 'combobox']); +const TEXT_INPUT_ROLES = new Set(['textbox', 'searchbox', 'combobox']); + +/** + * Whether a node may take typed text: a text input role, or an `other` (only + * iOS reports one, for an element it did not classify) that holds a value or + * nothing but text. agent-device 0.21.33's simulator AX tree reports a + * custom-drawn field (Compose Multiplatform, Flutter) that way: a filled one + * with its text as the value, an empty one over the placeholder it draws. + * A childless, valueless `other`, such as a React Native `Pressable`, and a + * container of controls stay refused. + */ +export function mayTakeText(node: SemanticNode): boolean { + if (node.role === undefined) return false; + if (TEXT_INPUT_ROLES.has(node.role)) return true; + if (node.role !== 'other') return false; + if (node.value !== undefined) return true; + const children = node.children ?? []; + return children.length > 0 && children.every((child) => child.role === 'text' && child.children === undefined); +} + const CHECKABLE_ROLES = new Set(['switch', 'checkbox']); const CHECKED_VALUES = new Set(['1', 'on', 'true', 'checked', 'selected']); const UNCHECKED_VALUES = new Set(['0', 'off', 'false', 'unchecked']); @@ -300,6 +318,31 @@ function showsHint(source: RawNode, android: boolean, editable: boolean): boolea return placeholder !== undefined && placeholder !== '' && source.value?.trim() === placeholder; } +/** + * The placeholder an empty iOS field folds into its label, with the child + * that draws it. Compose Multiplatform reports no placeholder: it draws one + * as a text inside the field and appends that text to the field's label, so + * an empty "Name" field with the example "Ada Lovelace" reads + * `"Name, Ada Lovelace"` over one `StaticText` child. Undefined unless the + * field's only child is that text and the field has a label of its own. + */ +function foldedPlaceholder( + source: RawNode, + kids: readonly number[], + raw: readonly RawNode[], + children: ReadonlyMap, +): { label: string; placeholder: string; child: number } | undefined { + if (kids.length !== 1) return undefined; + const child = kids[0] as number; + const text = raw[child] as RawNode; + if (children.has(child)) return undefined; + const placeholder = nonEmpty(text.label); + const label = source.label; + if (kindOf(text) !== 'static-text' || placeholder === undefined || label === undefined) return undefined; + if (!label.endsWith(`, ${placeholder}`) || label.length === placeholder.length + 2) return undefined; + return { label: label.slice(0, -(placeholder.length + 2)), placeholder, child }; +} + /** The string itself, or undefined for an absent or empty one. */ function nonEmpty(value: string | undefined): string | undefined { return value === undefined || value === '' ? undefined : value; @@ -450,9 +493,12 @@ export function projectSnapshot( ...(spelled?.states.expanded === undefined ? {} : { expanded: spelled.states.expanded }), }; const identifier = nonEmpty(source.identifier); - const label = nonEmpty(source.label); + const kids = children.get(position) ?? []; + const blank = editable && !secure && (value === undefined || value === '') && nonEmpty(source.placeholder) === undefined; + const folded = !android && blank ? foldedPlaceholder(source, kids, raw, children) : undefined; + const label = folded === undefined ? nonEmpty(source.label) : folded.label; const name = nonEmpty(source.contentDescription) ?? label; - const placeholder = nonEmpty(source.placeholder); + const placeholder = folded?.placeholder ?? nonEmpty(source.placeholder); const projected: { node: SemanticNode | undefined } = { node: undefined }; const entry: ProjectedNode = { id, @@ -465,7 +511,7 @@ export function projectSnapshot( }, }; index.push(entry); - const childNodes = (children.get(position) ?? []).map((child) => build(child, entry)); + const childNodes = kids.filter((child) => child !== folded?.child).map((child) => build(child, entry)); const node: SemanticNode = { ref: { id, revision: '' }, ...(role === undefined ? {} : { role }), diff --git a/packages/mobile/src/surface.ts b/packages/mobile/src/surface.ts index 1263f02df5..9d956c79fb 100644 --- a/packages/mobile/src/surface.ts +++ b/packages/mobile/src/surface.ts @@ -49,10 +49,10 @@ import { pointerInteraction, DEFAULT_LONG_PRESS_MS } from './actions.ts'; import { resolveExpression } from './locate.ts'; import { isWithin, + mayTakeText, normalizeKind, projectSnapshot, ROOT_ID, - TEXT_INPUT_ROLES, screenRoot, screenTitle, type ProjectedNode, @@ -201,6 +201,8 @@ interface NodeBinding { readonly node: SemanticNode; readonly controlRef: string; readonly keyboardVisible: boolean; + /** Whether `fill` and `clear` may type into the node, judged with the children `node` leaves out. */ + readonly takesText: boolean; } @@ -1161,7 +1163,14 @@ export class AgentDeviceSurface { private bind(entry: ProjectedNode, snapshot: ProjectedScreen): NodeBinding { const { children: _children, ...node } = entry.node; const control = this.controlOf(entry, snapshot.index); - return { id: entry.id, ref: entry.ref, node, controlRef: control.ref, keyboardVisible: snapshot.keyboardVisible }; + return { + id: entry.id, + ref: entry.ref, + node, + controlRef: control.ref, + keyboardVisible: snapshot.keyboardVisible, + takesText: mayTakeText(entry.node), + }; } /** @@ -1270,9 +1279,9 @@ export class AgentDeviceSurface { } const entry = this.resolveRef(ref); const filling = action.kind === 'fill' || action.kind === 'clear'; - if (filling && (entry.node.role === undefined || !TEXT_INPUT_ROLES.has(entry.node.role))) { - throw unsupported(`agent-device can only ${action.kind} editable fields; node ${entry.id} is ${entry.node.role ?? 'unknown'}`); - } + const refuseFill = (binding: NodeBinding): EngineError => + unsupported(`agent-device can only ${action.kind} editable fields; node ${binding.id} is ${binding.node.role ?? 'unknown'}`); + if (filling && !entry.takesText) throw refuseFill(entry); const label = `perform ${action.kind}`; const client = this.requireClient(); // A toggle already in the wanted state sends nothing, so it neither waits @@ -1286,6 +1295,7 @@ export class AgentDeviceSurface { if (filling) { if (deterministic) await this.waitForTransition(entry, operation); target = await this.relocated(entry, operation, true); + if (!target.takesText) throw refuseFill(target); } else if (deterministic) { target = await this.settled(entry, operation); } diff --git a/packages/mobile/tests/unit/engine.test.ts b/packages/mobile/tests/unit/engine.test.ts index 0a0e4b7033..0c6d5c28fa 100644 --- a/packages/mobile/tests/unit/engine.test.ts +++ b/packages/mobile/tests/unit/engine.test.ts @@ -2495,6 +2495,52 @@ describe('input freshness', () => { expect(h.fake.methods().filter((method) => method === 'capture.snapshot')).toHaveLength(captures); expect(h.fake.methods()).not.toContain('interactions.fill'); }); + + // agent-device 0.21.33's simulator AX tree reads a Compose Multiplatform field as an unclassified `Other`: + // an empty one over the placeholder it draws, a filled one with its text as the value. + const unclassified = (field: Partial, inside?: RawNode) => { + const snapshot = form(237, { type: 'Other', ...field }); + return inside === undefined ? snapshot : { ...snapshot, nodes: [...snapshot.nodes, inside] }; + }; + const child = (type: string, label: string): RawNode => + ({ ref: '@e22', index: 3, parentIndex: 1, depth: 2, type, label, rect: { x: 52, y: 252, width: 101, height: 18 } }); + + it.each([ + ['empty, over its placeholder', () => unclassified({}, child('StaticText', 'Ada Lovelace'))], + ['filled', () => unclassified({ value: 'Grace' })], + ])(`acts on an unclassified iOS field, %s, for ${origin} ${kind}`, async (_name, snapshot) => { + const h = harness(); + await openAttempt(h); + h.fake.respond('capture.snapshot', snapshot); + const field = await observed(h, 'Account'); + const action = kind === 'fill' ? { kind, value: 'example', sensitive: false } : { kind }; + await h.engine.perform!(field.ref, action, { ...operation(), origin }); + expect(h.fake.lastArgs('interactions.fill')).toMatchObject({ ref: '@e20', text: kind === 'fill' ? 'example' : '' }); + }); + + it.each([ + ['childless and valueless, like a React Native Pressable', () => unclassified({})], + ['holding a control', () => unclassified({}, child('Button', 'Clear'))], + ])(`rejects an unclassified iOS element %s for ${origin} ${kind}`, async (_name, snapshot) => { + const h = harness(); + await openAttempt(h); + h.fake.respond('capture.snapshot', snapshot); + const element = await observed(h, 'Account'); + const action = kind === 'fill' ? { kind, value: 'example', sensitive: false } : { kind }; + await expect(h.engine.perform!(element.ref, action, { ...operation(), origin })).rejects.toMatchObject({ code: 'UNSUPPORTED_CAPABILITY', retryable: false }); + expect(h.fake.methods()).not.toContain('interactions.fill'); + }); + + it(`rejects an unclassified iOS element that gained a control before typing for ${origin} ${kind}`, async () => { + const h = harness(); + await openAttempt(h); + h.fake.respond('capture.snapshot', () => unclassified({}, child('StaticText', 'Loading'))); + const element = await observed(h, 'Account'); + h.fake.respond('capture.snapshot', () => unclassified({}, child('Button', 'Loading'))); + const action = kind === 'fill' ? { kind, value: 'example', sensitive: false } : { kind }; + await expect(h.engine.perform!(element.ref, action, { ...operation(), origin })).rejects.toMatchObject({ code: 'UNSUPPORTED_CAPABILITY', retryable: false }); + expect(h.fake.methods()).not.toContain('interactions.fill'); + }); } } diff --git a/packages/mobile/tests/unit/nodes.test.ts b/packages/mobile/tests/unit/nodes.test.ts index 269eebad0e..eaebbf9ff8 100644 --- a/packages/mobile/tests/unit/nodes.test.ts +++ b/packages/mobile/tests/unit/nodes.test.ts @@ -437,6 +437,39 @@ describe('snapshot projection', () => { expect(android.index[0]?.node.states).toBeUndefined(); }); + it('reads the placeholder an empty Compose Multiplatform field folds into its label', () => { + // examples/with-kotlin-multiplatform on iOS 27 (agent-device 0.21.33's XCTest tree, Compose Multiplatform 1.12.1): + // the field has no placeholder attribute; it appends the text it draws to its label. + const field = (label: string, children: readonly RawNode[], extra: RawNode = {}) => project([ + { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'with-kotlin-multiplatform' }, + { ref: 'e2', index: 1, parentIndex: 0, depth: 1, type: 'TextView', label, identifier: 'name', editable: true, ...extra }, + ...children.map((child, offset) => ({ ref: `e${3 + offset}`, index: 2 + offset, parentIndex: 1, depth: 2, ...child })), + ]); + const hint = { type: 'StaticText', label: 'Ada Lovelace' }; + const empty = field('Name, Ada Lovelace', [hint]); + expect(empty.index.map((entry) => entry.node.name)).toEqual(['with-kotlin-multiplatform', 'Name']); + expect(empty.index[1]?.node).toMatchObject({ role: 'textbox', name: 'Name', attributes: { placeholder: 'Ada Lovelace' } }); + expect(empty.index[1]?.node.children).toBeUndefined(); + // Typed text drops the placeholder from the label and is the value. + const typed = field('Name', [], { value: 'Grace' }); + expect(typed.index[1]?.node).toMatchObject({ name: 'Name', value: 'Grace' }); + // Anything but one text that ends the label stays as reported. + for (const [label, children, extra] of [ + ['Name, Ada Lovelace', [hint], { value: 'Ada' }], + ['Name, Ada Lovelace', [hint], { editable: false, type: 'Other' }], + ['Name, Ada Lovelace', [hint, { type: 'Button', label: 'Clear' }], {}], + ['Name, Ada Lovelace', [{ type: 'Image', label: 'Ada Lovelace' }], {}], + ['Name, Grace Hopper', [hint], {}], + // A label equal to the text it holds may be the field's own name (a Material label drawn inside). + ['Ada Lovelace', [hint], {}], + ] as const) { + const kept = field(label, children, extra).index[1]?.node; + expect(kept?.name).toBe(label); + expect(kept?.attributes).toBeUndefined(); + expect(kept?.children).toHaveLength(children.length); + } + }); + it('keeps the value of an iOS field whose text is also its label, and drops only the Android text view echo', () => { const projected = project([ { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'Benchmark' }, diff --git a/plugins/e2e/skills/e2e/references/setup.md b/plugins/e2e/skills/e2e/references/setup.md index b56ad36884..6d765ce7d4 100644 --- a/plugins/e2e/skills/e2e/references/setup.md +++ b/plugins/e2e/skills/e2e/references/setup.md @@ -383,7 +383,9 @@ export default { - Mobile `fill` and `clear` re-read the field before typing, replay included. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation. - Non-input targets return `UNSUPPORTED_CAPABILITY`. + Non-input targets return `UNSUPPORTED_CAPABILITY`; an iOS `other` that holds + a value or only text is typed into, since a custom-drawn field (Compose + Multiplatform, Flutter) can read that way. - `screen`, `expect`, `app`, `agent` work unchanged; `test` from `@e2e-dev/mobile` types the `device` fixture (`installApp`, `openLink`, `setPermission`, `setNetwork`, `setAppearance`, `clearKeychain`, `fold` diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index b56ad36884..6d765ce7d4 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -383,7 +383,9 @@ export default { - Mobile `fill` and `clear` re-read the field before typing, replay included. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation. - Non-input targets return `UNSUPPORTED_CAPABILITY`. + Non-input targets return `UNSUPPORTED_CAPABILITY`; an iOS `other` that holds + a value or only text is typed into, since a custom-drawn field (Compose + Multiplatform, Flutter) can read that way. - `screen`, `expect`, `app`, `agent` work unchanged; `test` from `@e2e-dev/mobile` types the `device` fixture (`installApp`, `openLink`, `setPermission`, `setNetwork`, `setAppearance`, `clearKeychain`, `fold`