diff --git a/.changeset/mobile-gesture-rest-wait.md b/.changeset/mobile-gesture-rest-wait.md new file mode 100644 index 0000000000..ba971ca8d8 --- /dev/null +++ b/.changeset/mobile-gesture-rest-wait.md @@ -0,0 +1,5 @@ +--- +"@e2e-dev/mobile": patch +--- + +`mobile({ transition })` takes an object, `{ default, gesture, launch }` (the new `TransitionBudgets` type), to set the wait before acting on a control by the kind of the last action. A number still sets the same wait after every action. After a scroll or swipe, a test now waits for a control that came with it to stop moving: the `gesture` budget is the least wait, then the control is read until two reads agree. The wait ends at the latest 4 s after the gesture (the `gesture` budget when that is larger), or sooner when the action's timeout cannot fit another read, and the action then goes ahead on the latest read. A budget of 0 is still no wait at all. diff --git a/apps/mobile-benchmark/e2e.config.ts b/apps/mobile-benchmark/e2e.config.ts index a5ee78d874..b6c7afc7a4 100644 --- a/apps/mobile-benchmark/e2e.config.ts +++ b/apps/mobile-benchmark/e2e.config.ts @@ -36,9 +36,10 @@ export const ios = mobile({ session: 'e2e-mobile-benchmark-ios', // A control that just moved waits this long to come to rest before a tap; // an unmoved one is tapped at once, so the budget costs only after a - // scroll. iOS reports a row's final frame from the first frame of a fling, - // and the default half second let a tap after scrollUntilVisible land on - // the row still passing underneath on a loaded simulator. + // scroll. The default half second let a tap after scrollUntilVisible land + // on the row still passing underneath on a loaded simulator. After a scroll + // the engine also reads until the control stops moving, so this budget is + // the least wait. transition: 1_500, }); diff --git a/docs/reference/mobile.mdx b/docs/reference/mobile.mdx index 75f3fbf9e8..e8a6d43c09 100644 --- a/docs/reference/mobile.mdx +++ b/docs/reference/mobile.mdx @@ -85,7 +85,7 @@ The app under test is the target's `app`, not an engine option; see | [`session`](#session) | `string` | `e2e-` | | [`snapshot`](#snapshot) | `'full' \| 'interactive'` | `'full'` | | [`settle`](#settle) | `number \| false` | `150` | -| [`transition`](#transition) | `number` | `500` | +| [`transition`](#transition) | `number \| TransitionBudgets` | `500` | | [`videoTouches`](#videotouches) | `boolean` | `true` | Every optional value also accepts `undefined`, so values read straight from @@ -99,7 +99,8 @@ declaration. | `INVALID_CONFIG` | `platform` is missing or not `'ios'` or `'android'`; the match is case-sensitive. | | `INVALID_CONFIG` | A provided `snapshot` value is not `'full'` or `'interactive'`, or a provided `videoTouches` value is not a boolean. | | `INVALID_CONFIG` | An option `mobile()` does not take, such as `mobile({ app })` and the other old app options. The message names the nearest known option for a likely typo, else every option. | -| `INVALID_CONFIG` | `settle` or `transition` is not a non-negative integer of milliseconds (`settle` may also be `false`). | +| `INVALID_CONFIG` | `settle` or `transition` is not a non-negative integer of milliseconds. `settle` may also be `false`, and `transition` an object of budgets. | +| `INVALID_CONFIG` | A budget in a `transition` object is not a non-negative integer of milliseconds, or the object has a key other than `default`, `gesture`, and `launch`. The message names the budget, or the nearest known key for a likely typo, else every key. | #### `platform` @@ -232,31 +233,74 @@ mobile({ platform: 'ios', settle: 300 }); // default: 150 ```ts /** - * Wait before acting on a control that just appeared or moved, in - * milliseconds. Default 500. Accessibility bounds cannot tell a sliding - * control from a landed one (iOS reports the final frame at once, Android - * the frame in flight), so the control is found again in a fresh snapshot - * once the wait is over and acted on where it landed. Unmoved controls are - * acted on immediately only when keyboard visibility is unchanged. Opening - * or closing the keyboard also uses the remaining budget. Screen swipes in - * tests wait out the remaining budget before scrolling. Raise it for slower - * transitions. + * How long a control that appeared or moved with the last action is given + * to finish arriving before a test acts on it, in milliseconds. Default + * 500, the length of a modal or screen transition. Accessibility bounds + * cannot tell a sliding control from a landed one (iOS reports the final + * frame at once, Android the frame in flight), so the control is found + * again in a fresh snapshot once the wait is over and acted on where it + * landed. Unmoved controls are acted on immediately only when keyboard + * visibility is unchanged. Opening or closing the keyboard also uses the + * remaining budget. Screen swipes in tests wait out the remaining budget + * before scrolling. Raise it for slower transitions. A number is that + * budget after every action. An object sets it by the kind of the last + * action: `default` for any action except a gesture or a launch, `gesture` + * after a scroll or swipe, `launch` after the app launches, relaunches, or + * closes, or a link opens into it. A `gesture` or `launch` budget left out + * takes the `default` one. A scroll or swipe can return while its content + * still moves, and the tree shows that content in flight, so the `gesture` + * budget is only the least wait. A control that arrived or moved with the + * gesture is then read until it stops moving, which is two reads that + * agree on where it and any control that looks like it sit, at most 4 s + * after the gesture, or at the budget when that is larger, or sooner when + * the action's own timeout cannot fit another read. A plain number sets the + * `gesture` budget too. A budget of 0 is no wait at all. After any other + * action the budget is the whole wait. */ -readonly transition?: number | undefined; +readonly transition?: number | TransitionBudgets | undefined; ``` ```ts mobile({ platform: 'ios', transition: 800 }); // default: 500 -``` +mobile({ platform: 'ios', transition: { default: 750, gesture: 1_500, launch: 500 } }); +``` + +An object sets the wait by what the last action was. `gesture` is the wait +after a scroll or swipe, `launch` after the app launches, relaunches, or +closes, or a link opens into it, and `default` after any other action. +`default` is 500 unless set, and `gesture` and `launch` take the `default` +budget when left out, so `{ gesture: 1_500 }` waits 1.5 s after a scroll or +swipe and 500 ms after everything else. Every budget is a non-negative integer +of milliseconds, and a key other than these three is `INVALID_CONFIG`. + +The `gesture` budget is the least wait, not the whole of it. A scroll or swipe +can return while its content is still moving, and the tree shows that content +in flight, so the wait goes on until the control stops. After any other action +the budget is the whole wait, since the tree cannot tell a control that slides +in from one that landed. A plain number sets the `gesture` budget too. A +budget of 0 is no wait at all, so `transition: 0` and `{ gesture: 0 }` act at +once. + +After the `gesture` budget, the engine reads the screen until a read agrees +with the one before it, or with the one that found the control. Two reads agree +when the control and its look-alikes are the same in number and sit at the same +places. A look-alike has the same role, test ID, name, and text. The wait ends +at most 4 s after the gesture, or at the budget when that is larger, and the +action then goes ahead where the control was last read. A control that never +stops moving costs the cap, 4 s or the `gesture` budget when that is larger, or +what the action's own timeout leaves: the wait ends once the timeout cannot fit +another read, and the action goes ahead where the control was last read. If the control changes during the wait, the action finds it again within its timeout before acting. `back`, `home`, `alert`, `dismissKeyboard`, `setOrientation`, and `fold` on [`device`](#device) change the screen, so they count as actions for this -budget the way a tap does. A control that arrives with the screen they reveal -waits out the budget before the next action lands on it. +budget the way a tap does, and the `default` budget applies after them. A +control that arrives with the screen they reveal waits out the budget before +the next action lands on it. -`fill` and `clear` re-read the field before typing. Missing fields, changed +`fill` and `clear` re-read the field before typing; after a scroll or swipe, the +read that found the field at rest is that re-read. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation. Non-input targets return `UNSUPPORTED_CAPABILITY`. An iOS `other`, the type of an element the platform @@ -1190,6 +1234,26 @@ export type PermissionState = 'grant' | 'deny' | 'reset'; /** Permissions an app holds when it launches, by name. */ export type LaunchPermissions = Readonly>>; + +/** + * The `transition` budgets by the kind of the last action, in milliseconds, + * each a non-negative integer. A `gesture` or `launch` budget left out takes + * the `default` one. + */ +export interface TransitionBudgets { + /** After any action except a gesture or a launch: a tap, fill, key, back, and the rest. Default 500. */ + readonly default?: number | undefined; + /** + * After a scroll or swipe, which can return while the content is still + * moving. The least wait. The wait then goes on until the control stops + * moving, two reads agreeing, at most 4 s after the gesture. A budget above + * 4 s ends the wait at that budget, and 0 is no wait at all. Default: the + * `default` budget. + */ + readonly gesture?: number | undefined; + /** After the app launches, relaunches, or closes, or a link opens into it. Default: the `default` budget. */ + readonly launch?: number | undefined; +} ``` ### Device fixture @@ -1289,7 +1353,7 @@ export interface DeviceSelection { ### Exported types `MobileOptions`, `MobilePlatform`, `DevicePermission`, `LaunchPermissions`, -`PermissionState`, `DeviceClientConfig`, `DeviceConnection`, `DeviceDaemon`, +`PermissionState`, `TransitionBudgets`, `DeviceClientConfig`, `DeviceConnection`, `DeviceDaemon`, `DeviceProvider`, `DeviceLease`, `DeviceReleaseContext`, `DeviceRequest`, `Device`, `DeviceSession`, `DeviceSessionClient`, `DeviceSelection`, `DeviceOrientation`, `BiometricSensor`, `ForegroundApp`, `FoldPose`, `InstallAppOptions`, `InstalledApp`, and `OpenAppOptions` are exported from diff --git a/packages/mobile/README.md b/packages/mobile/README.md index 9e9ebbdc83..9693afeaec 100644 --- a/packages/mobile/README.md +++ b/packages/mobile/README.md @@ -73,7 +73,7 @@ the other old app options are `INVALID_CONFIG` naming their key under `app`. | `session` | agent-device session name, before the worker slot: slot `n` drives its device under `-`, `e2e--` by default. One run per session at a time. | | `snapshot` | `'full'` (default, includes static text) or `'interactive'` (actionable nodes only). | | `settle` | Milliseconds the UI must hold still after an agent action before the agent observes again, default `150`; `false` skips the wait. A test's steps do not settle, except `device.back()`; `expect` verifies their outcome. | -| `transition` | For a test's steps: milliseconds a control that appeared or moved with the last action gets to finish arriving before it is acted on, default `500`. Unmoved controls are acted on immediately only when keyboard visibility is unchanged. Opening or closing the keyboard also uses the remaining budget. Screen swipes in tests wait out the remaining budget before scrolling. | +| `transition` | For a test's steps: milliseconds a control that appeared or moved with the last action gets to finish arriving before it is acted on, default `500`. An object sets it by the kind of the last action: `default` for any action except a gesture or a launch, `gesture` after a scroll or swipe, `launch` after the app launches, relaunches, or closes, or a link opens into it. A `gesture` or `launch` budget left out takes `default`. The `gesture` budget is the least wait. The control is then read until it stops moving, which is two reads that agree, at most 4 s after the gesture, or at the budget when that is larger, or sooner when the action's own timeout cannot fit another read. A plain number sets it too, and 0 is no wait at all. Unmoved controls are acted on immediately only when keyboard visibility is unchanged. Opening or closing the keyboard also uses the remaining budget. Screen swipes in tests wait out the remaining budget before scrolling. | | `videoTouches` | For recordings the mobile engine makes (not a device provider's own recording), whether the video draws agent-device's touch indicator where a test tapped, default `true`. `false` records the plain screen and skips drawing the indicator after the recording stops, which on a hosted daemon can outlast `cleanupTimeout`. | Every optional `mobile()` value also accepts `undefined`, so a config passes diff --git a/packages/mobile/src/index.ts b/packages/mobile/src/index.ts index 61440f0ec0..b96079edc4 100644 --- a/packages/mobile/src/index.ts +++ b/packages/mobile/src/index.ts @@ -12,7 +12,7 @@ import type { Device } from './device.ts'; export { mobile } from './engine.ts'; /** The engine factory again, as the default export: what `e2e mcp` calls when `open_session` names this package as its `engine`. */ export { mobile as default } from './engine.ts'; -export type { DevicePermission, LaunchPermissions, MobileOptions, MobilePlatform, PermissionState } from './options.ts'; +export type { DevicePermission, LaunchPermissions, MobileOptions, MobilePlatform, PermissionState, TransitionBudgets } from './options.ts'; export type { DeviceClientConfig, DeviceConnection, DeviceDaemon } from './bindings.ts'; export type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from './provider.ts'; export type { FoldPose, InstallAppOptions, InstalledApp, OpenAppOptions } from './surface.ts'; diff --git a/packages/mobile/src/options.ts b/packages/mobile/src/options.ts index 4b497e8a7a..fc079cdfbb 100644 --- a/packages/mobile/src/options.ts +++ b/packages/mobile/src/options.ts @@ -43,6 +43,26 @@ export type PermissionState = 'grant' | 'deny' | 'reset'; /** Permissions an app holds when it launches, by name. */ export type LaunchPermissions = Readonly>>; +/** + * The `transition` budgets by the kind of the last action, in milliseconds, + * each a non-negative integer. A `gesture` or `launch` budget left out takes + * the `default` one. + */ +export interface TransitionBudgets { + /** After any action except a gesture or a launch: a tap, fill, key, back, and the rest. Default 500. */ + readonly default?: number | undefined; + /** + * After a scroll or swipe, which can return while the content is still + * moving. The least wait. The wait then goes on until the control stops + * moving, two reads agreeing, at most 4 s after the gesture. A budget above + * 4 s ends the wait at that budget, and 0 is no wait at all. Default: the + * `default` budget. + */ + readonly gesture?: number | undefined; + /** After the app launches, relaunches, or closes, or a link opens into it. Default: the `default` budget. */ + readonly launch?: number | undefined; +} + /** * Options of the device engine: which devices it drives and how. The app * under test is the target's `app` (`bundleId`, `appPath`, @@ -107,15 +127,29 @@ export interface MobileOptions { /** * How long a control that appeared or moved with the last action is given * to finish arriving before a test acts on it, in milliseconds. Default - * 500, the length of a modal or screen transition. Accessibility frames - * report a control's final position from the first frame of a transition, - * so this budget is the only thing that keeps a test from tapping a point - * the control has not reached. Unmoved controls are acted on immediately - * only when keyboard visibility is unchanged. Opening or closing the - * keyboard also uses the remaining budget. Screen swipes in tests wait out - * the remaining budget before scrolling. Raise it for slower transitions. + * 500, the length of a modal or screen transition. Accessibility bounds + * cannot tell a sliding control from a landed one (iOS reports the final + * frame at once, Android the frame in flight), so the control is found + * again in a fresh snapshot once the wait is over and acted on where it + * landed. Unmoved controls are acted on immediately only when keyboard + * visibility is unchanged. Opening or closing the keyboard also uses the + * remaining budget. Screen swipes in tests wait out the remaining budget + * before scrolling. Raise it for slower transitions. A number is that + * budget after every action. An object sets it by the kind of the last + * action: `default` for any action except a gesture or a launch, `gesture` + * after a scroll or swipe, `launch` after the app launches, relaunches, or + * closes, or a link opens into it. A `gesture` or `launch` budget left out + * takes the `default` one. A scroll or swipe can return while its content + * still moves, and the tree shows that content in flight, so the `gesture` + * budget is only the least wait. A control that arrived or moved with the + * gesture is then read until it stops moving, which is two reads that + * agree on where it and any control that looks like it sit, at most 4 s + * after the gesture, or at the budget when that is larger, or sooner when + * the action's own timeout cannot fit another read. A plain number sets the + * `gesture` budget too. A budget of 0 is no wait at all. After any other + * action the budget is the whole wait. */ - readonly transition?: number | undefined; + readonly transition?: number | TransitionBudgets | undefined; /** * Whether a video recording draws agent-device's touch indicator where a * test tapped. Default `true`. `false` records the plain screen, and skips diff --git a/packages/mobile/src/surface.ts b/packages/mobile/src/surface.ts index 1a36fe7bd5..30a4849097 100644 --- a/packages/mobile/src/surface.ts +++ b/packages/mobile/src/surface.ts @@ -63,7 +63,16 @@ import { type ProjectedSnapshot, } from './nodes.ts'; import { keyboardDragStart } from './keyboard.ts'; -import { DEVICE_PERMISSIONS, type AgentDeviceClient, type ClientFactory, type DevicePermission, type LaunchPermissions, type MobileOptions, type PermissionState } from './options.ts'; +import { + DEVICE_PERMISSIONS, + type AgentDeviceClient, + type ClientFactory, + type DevicePermission, + type LaunchPermissions, + type MobileOptions, + type PermissionState, + type TransitionBudgets, +} from './options.ts'; import { maskPng } from './png.ts'; import { mobileDoctor } from './doctor.ts'; import { deviceLabel, pinnedApp, type SlotBinding } from './bindings.ts'; @@ -220,20 +229,52 @@ interface NodeBinding { * a tap aimed from either frame lands on whatever is behind the control. * Once the budget has passed the control is found again in a fresh * snapshot. Unmoved controls are immediate only when keyboard state also - * stayed the same; a visible keyboard can precede the app's relayout. + * stayed the same; a visible keyboard can precede the app's relayout. This is + * the budget after every action when the `transition` option sets none; the + * option can give a gesture or a launch a budget of its own, see + * {@link TransitionBudgets}. A scroll or swipe is a different case from a + * control that slides in. It can return while its content still moves, and the + * tree shows that content in flight. After a gesture the budget is only the + * least wait, and the wait goes on until the control is seen at rest, see + * GESTURE_REST_CAP_MS. */ const DEFAULT_TRANSITION_MS = 500; /** - * How long after a test's swipe or scroll skipped agent-device's stabilization - * wait the next action on a control first confirms the control stopped moving. - * A list still coasting swallows a tap to stop itself, so the press would - * report success and change nothing. + * The longest a control that came with a gesture is waited for to stop + * moving, counted from when the gesture returned. A `gesture` budget above it + * is the cap instead. A scroll can return while its fling goes on, and the + * tree shows the content in flight. On an iOS simulator with agent-device + * 0.21.36, 3 of 120 scrolls of a long list left it moving 1.66 to 1.84 s after + * `scroll` returned, and the reads taken meanwhile saw 2 to 3 different + * positions. Past the cap the action goes ahead where the control was last + * read, as it would after a blind sleep. A screen that never stops moving + * costs the cap and no more. + */ +const GESTURE_REST_CAP_MS = 4_000; + +/** + * The pause between two reads of a control that has not come to rest. A read + * takes time of its own on a device. The pause keeps a daemon that answers at + * once from being read in a tight loop until the cap. + */ +const REST_READ_GAP_MS = 50; + +/** The kind of an action, which picks the transition budget after it: `gesture` for a scroll or swipe, `launch` for the app starting or closing, `default` for the rest. */ +type ActionKind = keyof TransitionBudgets; + +/** The transition budget after each kind of action, in milliseconds. */ +type TransitionWaits = Readonly>; + +/** + * What a transition wait leaves its caller: `false` when it waited for + * nothing, `true` after a sleep that the control must be read again from, or + * the control itself, as the read that found it at rest saw it. */ -const GESTURE_STEADY_MS = 2_000; +type TransitionWait = boolean | NodeBinding; -/** Pause between the reads that confirm a control stopped moving after a gesture. */ -const GESTURE_STEADY_POLL_MS = 100; +/** Where a control and every node that looks like it sat in one capture, top to bottom, then left to right. */ +type Sighting = readonly Rect[]; /** How one screenshot is taken. */ interface ScreenshotCapture { @@ -258,6 +299,37 @@ function sameRect(a: Rect | undefined, b: Rect | undefined): boolean { return Math.abs(a.x - b.x) < 1 && Math.abs(a.y - b.y) < 1 && Math.abs(a.width - b.width) < 1 && Math.abs(a.height - b.height) < 1; } +/** The nodes of a capture that look like a binding: the same role, test id, name, and text. The binding's own node is one of them. */ +function lookalikes(entry: NodeBinding, index: readonly ProjectedNode[]): ProjectedNode[] { + return index.filter( + (candidate) => + candidate.node.role === entry.node.role && + candidate.node.testId === entry.node.testId && + candidate.node.name === entry.node.name && + candidate.node.text === entry.node.text, + ); +} + +/** + * Where a binding's control and its look-alikes sit in one capture; undefined + * when one has no bounds, since nothing can be said to hold still then. The + * look-alikes count so that a row of a repeated list, moved by exactly one row + * between two reads, does not pass for one that stood still. + */ +function sighting(entry: NodeBinding, index: readonly ProjectedNode[]): Sighting | undefined { + const rects: Rect[] = []; + for (const candidate of lookalikes(entry, index)) { + if (candidate.node.rect === undefined) return undefined; + rects.push(candidate.node.rect); + } + return rects.toSorted((a, b) => a.y - b.y || a.x - b.x); +} + +/** Whether two sightings are one: as many nodes, each in the same place as its counterpart in position order. */ +function sameSighting(a: Sighting | undefined, b: Sighting | undefined): boolean { + return a !== undefined && b !== undefined && a.length === b.length && a.every((rect, position) => sameRect(rect, b[position])); +} + const MAX_LOCATED_REFS = 2048; interface Attempt { @@ -398,11 +470,13 @@ function permissionError(code: 'INVALID_CONFIG' | 'INVALID_ARGUMENT', message: s throw code === 'INVALID_CONFIG' ? new ConfigurationError(code, message) : new TestError(code, message); } -/** Resolves the `transition` option: the default budget or a custom one. */ -function transitionMs(transition: MobileOptions['transition']): number { - const budget = transition ?? DEFAULT_TRANSITION_MS; +/** Every budget `transition` takes by kind of action, kept equal to `TransitionBudgets` by the compiler. */ +const TRANSITION_BUDGET_KEYS: readonly string[] = Object.keys({ default: true, gesture: true, launch: true } satisfies Record); + +/** Refuses a budget that is not a non-negative integer of milliseconds; `option` is how the message names it. */ +function budgetMs(budget: number, option: string): number { if (!Number.isInteger(budget) || budget < 0) { - throw new ConfigurationError('INVALID_CONFIG', 'mobile: `transition` must be a non-negative integer of milliseconds'); + throw new ConfigurationError('INVALID_CONFIG', `mobile: \`${option}\` must be a non-negative integer of milliseconds`); } return budget; } @@ -412,10 +486,33 @@ function gestureStabilization(operation: OperationContext): GestureStabilization return operation.origin === 'test' ? { postGestureStabilization: false } : {}; } -/** True when two bounds are the same; a control without bounds has nothing to compare, so it counts as unmoved. */ -function sameBounds(a: NodeBinding['node']['rect'], b: NodeBinding['node']['rect']): boolean { - if (a === undefined || b === undefined) return true; - return a.x === b.x && a.y === b.y && a.width === b.width && a.height === b.height; +/** + * Resolves the `transition` option into the budget after each kind of action. + * A number, or no option, is one budget for all of them. An object sets a + * budget per kind, and `gesture` and `launch` take the `default` one when + * left out. + */ +function transitionWaits(transition: MobileOptions['transition']): TransitionWaits { + const budgets = transition ?? DEFAULT_TRANSITION_MS; + if (typeof budgets === 'number') { + const budget = budgetMs(budgets, 'transition'); + return { default: budget, gesture: budget, launch: budget }; + } + // A string, array, or class instance holds no budgets, so it would run on the default instead of failing. + const prototype: unknown = Object.getPrototypeOf(budgets); + if (prototype !== Object.prototype && prototype !== null) { + throw new ConfigurationError( + 'INVALID_CONFIG', + `mobile: \`transition\` must be a non-negative integer of milliseconds, or an object of them by kind of action (${TRANSITION_BUDGET_KEYS.join(', ')})`, + ); + } + rejectUnknownKeys('mobile: `transition`', budgets, TRANSITION_BUDGET_KEYS); + const base = budgetMs(budgets.default ?? DEFAULT_TRANSITION_MS, 'transition.default'); + return { + default: base, + gesture: budgetMs(budgets.gesture ?? base, 'transition.gesture'), + launch: budgetMs(budgets.launch ?? base, 'transition.launch'), + }; } /** Resolves the `settle` option: the default window, a custom one, or no wait at all. */ @@ -506,14 +603,14 @@ export class AgentDeviceSurface { readonly settleOptions: SettleOptions; /** When this surface last acted on the device (an input or an app launch); the screen may be in transition for a while after. */ private lastActionAt = 0; - /** When a test's swipe or scroll last went out without agent-device's stabilization wait; 0 when nothing is pending. */ - private skippedStabilizationAt = 0; + /** The kind of that action, which picks the budget the screen gets to come to rest. */ + private lastActionKind: ActionKind = 'default'; /** The screen as last projected before that action, to tell controls that were already there from ones that came with it. */ private screenBeforeAction: ProjectedScreen | undefined; /** The most recent projection of the screen, from any observe or locate. */ private latestScreen: ProjectedScreen | undefined; - /** Budget a control that came with the last action gets to finish arriving; see DEFAULT_TRANSITION_MS. */ - private readonly transitionMs: number; + /** Budget a control that came with the last action gets to finish arriving, by the kind of that action; see DEFAULT_TRANSITION_MS. */ + private readonly transitionWaits: TransitionWaits; /** Whether snapshots keep interactive nodes only, from the `snapshot` option. */ private readonly interactiveSnapshots: boolean; /** Whether recordings hide the touch overlay, from the `videoTouches` option. */ @@ -535,7 +632,7 @@ export class AgentDeviceSurface { rejectUnknownKeys('mobile()', options, MOBILE_OPTION_KEYS); this.pool = new DevicePool(options, createClient); this.settleOptions = settleOptions(options.settle); - this.transitionMs = transitionMs(options.transition); + this.transitionWaits = transitionWaits(options.transition); this.interactiveSnapshots = interactiveSnapshots(options.snapshot); this.hideTouches = hideTouches(options.videoTouches); } @@ -976,7 +1073,7 @@ export class AgentDeviceSurface { * bindings names a node that still exists. */ private screenReplaced(): void { - this.markAction(undefined); + this.markAction(undefined, 'launch'); this.generation = new Map(); this.located.clear(); } @@ -1305,13 +1402,7 @@ export class AgentDeviceSurface { * id, same name and text, and of those the one closest to where it was. */ private refind(entry: NodeBinding, index: readonly ProjectedNode[]): ProjectedNode | undefined { - const candidates = index.filter( - (candidate) => - candidate.node.role === entry.node.role && - candidate.node.testId === entry.node.testId && - candidate.node.name === entry.node.name && - candidate.node.text === entry.node.text, - ); + const candidates = lookalikes(entry, index); if (candidates.length <= 1 || entry.node.rect === undefined) return candidates[0]; const was = centreOf(entry.node.rect); let best = candidates[0]; @@ -1328,10 +1419,13 @@ export class AgentDeviceSurface { return best; } - /** Waits for the transition without exceeding the action's remaining budget. */ + /** + * Waits out what is left of the last action's budget, by the kind of that + * action, without exceeding the action's own remaining budget. + */ private async waitOutTransition(operation: OperationContext, budget: InputBudget): Promise { await raceAbort(async () => { - const remaining = this.transitionMs - (Date.now() - this.lastActionAt); + const remaining = this.transitionWaits[this.lastActionKind] - (Date.now() - this.lastActionAt); if (remaining <= 0) return; const available = budget.remainingMs(); if (available > 0) await sleep(Math.min(remaining, available), operation.signal); @@ -1344,57 +1438,110 @@ export class AgentDeviceSurface { } /** - * Waits for a control that arrived or moved with the last action, returning - * whether it needs a fresh capture. Controls already present at the same - * place need no wait, unless keyboard state changed: a keyboard change can - * precede the app's relayout. + * Waits for a control that arrived or moved with the last action until the + * budget of that action's kind has passed. Controls already present at the + * same place need no wait, unless keyboard state changed: a keyboard change + * can precede the app's relayout. + * + * After a gesture the budget is only the least wait. The control is then + * read until it is seen at rest, see {@link atRest}. After any other action + * the wait is a sleep, and a read after it finds the control again. No read + * can end that sleep early, since the tree cannot tell a control that is + * still sliding in from one that landed, see DEFAULT_TRANSITION_MS. A budget + * of 0 is no wait at all, after a gesture too: no sleep and no read. With + * `input`, the read that finds the control at rest also passes the checks + * `relocated` makes for an input, so a fill or clear needs no read of its own + * after it. `holdMs` is how long the action then holds, a long press, which + * `InputBudget.send` refuses unsent when less than that is left. */ - private async waitForTransition(entry: NodeBinding, operation: OperationContext, budget: InputBudget): Promise { + private async waitForTransition(entry: NodeBinding, operation: OperationContext, budget: InputBudget, input = false, holdMs = 0): Promise { const since = Date.now() - this.lastActionAt; - const remaining = this.transitionMs - since; - if (remaining <= 0) return false; + const kind = this.lastActionKind; + const floor = this.transitionWaits[kind]; + const until = kind === 'gesture' && floor > 0 ? Math.max(floor, GESTURE_REST_CAP_MS) : floor; + if (since >= until) return false; const before = this.screenBeforeAction; if (before !== undefined && before.keyboardVisible === entry.keyboardVisible && this.samePlace(entry, before.index) !== undefined) return false; - await this.waitOutTransition(operation, budget); - return true; - } - - /** Records that a test's gesture skipped the stabilization wait, so the next action on a control checks it stopped moving. */ - private noteGesture(operation: OperationContext): void { - if (operation.origin === 'test') this.skippedStabilizationAt = Date.now(); + if (since < floor) await this.waitOutTransition(operation, budget); + if (kind !== 'gesture') return true; + return this.atRest(entry, operation, budget, this.lastActionAt + until, input, holdMs); } /** - * After a gesture that skipped the stabilization wait, re-reads the control - * until two reads agree on where it is, so an action does not land on a list - * still coasting. Costs one extra snapshot when the screen already rests, - * and nothing once GESTURE_STEADY_MS has passed. An input action keeps the - * checks `relocated` makes on every read, and the poll never outlasts the - * action's budget. + * Reads the screen until the control is seen at rest, and returns it as that + * read saw it, so the caller needs no read of its own. Two sightings that + * agree are a rest. A sighting is the control and its look-alikes, and two + * agree when they are as many and sit at the same places. The pair is the + * read `entry` came from and the first read here, or two reads here in a + * row. The control is tracked through the reads as `refind` does. A read + * that no longer lists it is a retryable NODE_STALE, as in `relocated`, and + * not a wait for the cap. At `capAt` the latest read stands, agreeing or not. + * So does it when the action's own budget cannot fit another read and the + * pause before it, and the `holdMs` the action holds after them, as + * `snapshot` returns a sparse capture rather than time out on a retry it + * cannot fit. With `input`, the read it returns must also pass the checks of + * an input mutation, see {@link refuseChangedInput}. */ - private async steadied(entry: NodeBinding, operation: OperationContext, budget: InputBudget, input: boolean): Promise { - const deadline = this.skippedStabilizationAt + GESTURE_STEADY_MS; - if (this.skippedStabilizationAt === 0 || Date.now() >= deadline) return entry; - let current = entry; + private async atRest(entry: NodeBinding, operation: OperationContext, budget: InputBudget, capAt: number, input: boolean, holdMs: number): Promise { + let tracked = entry; + let previous = this.sightingAt(entry); for (;;) { - const next = await this.relocated(current, operation, budget, input); - const steady = sameBounds(current.node.rect, next.node.rect); - current = next; - if (steady || Date.now() >= deadline) break; - if (budget.remainingMs() <= GESTURE_STEADY_POLL_MS) { - throw new EngineError('OPERATION_TIMEOUT', 'the control kept moving until the action budget ran out', { retryable: false }); + const startedAt = Date.now(); + const raw = await this.snapshotOrEmpty(operation, false, budget); + const readMs = Date.now() - startedAt; + const projected = this.project(raw); + const found = this.refind(tracked, projected.index); + if (found === undefined) { + throw new EngineError('NODE_STALE', `node ${entry.id} changed during transition; re-resolve`, { retryable: true }); + } + tracked = this.bind(found, projected); + const seen = sighting(tracked, projected.index); + const left = capAt - Date.now(); + if (sameSighting(previous, seen) || left <= 0 || budget.remainingMs() <= REST_READ_GAP_MS + readMs + holdMs) { + if (input) this.refuseChangedInput(entry, found, raw); + return tracked; + } + previous = seen; + await sleep(Math.min(REST_READ_GAP_MS, left), operation.signal); + if (operation.signal.aborted) { + throw new EngineError('CANCELLED', 'transition wait cancelled', { retryable: false }); } - await sleep(GESTURE_STEADY_POLL_MS, operation.signal); - if (operation.signal.aborted) throw new EngineError('CANCELLED', 'gesture wait cancelled', { retryable: false }); } - this.skippedStabilizationAt = 0; - return current; } - /** Refreshes a control after a transition wait, keeping the binding when no wait was needed. */ - private async settled(entry: NodeBinding, operation: OperationContext, budget: InputBudget): Promise { - if (await this.waitForTransition(entry, operation, budget)) return this.relocated(entry, operation, budget); - return entry; + /** + * Where the control and its look-alikes sat in the read an action resolved + * on. That read is the newest projection when `entry` is one of its nodes. + * After a newer read only the control's own place is left of it, which a read + * with a look-alike never matches. + */ + private sightingAt(entry: NodeBinding): Sighting | undefined { + const latest = this.latestScreen; + if (latest?.index.some((node) => node.id === entry.id) === true) return sighting(entry, latest.index); + return entry.node.rect === undefined ? undefined : [entry.node.rect]; + } + + /** Refreshes a control after a transition wait. That is the control as the wait read it at rest, else found again after a sleep, else the binding itself when no wait was needed. */ + private async settled(entry: NodeBinding, operation: OperationContext, budget: InputBudget, holdMs = 0): Promise { + const waited = await this.waitForTransition(entry, operation, budget, false, holdMs); + if (waited === true) return this.relocated(entry, operation, budget); + return waited === false ? entry : waited; + } + + /** + * Refuses an input mutation when a fresh read shows the control changed from + * the binding the action resolved on: a truncated read, the control gone, or + * its input purpose, disabled, secure, or hidden state different. A retryable + * NODE_STALE, so the locator resolves and authorizes it again. + */ + private refuseChangedInput(entry: NodeBinding, found: ProjectedNode | undefined, raw: RawSnapshot): void { + if (isTruncated(raw) || found === undefined || + entry.node.inputPurpose !== found.node.inputPurpose || + entry.node.states?.disabled !== found.node.states?.disabled || + entry.node.states?.secure !== found.node.states?.secure || + entry.node.states?.hidden !== found.node.states?.hidden) { + throw new EngineError('NODE_STALE', `input node ${entry.id} changed before mutation; observe and authorize it again`, { retryable: true }); + } } /** @@ -1406,13 +1553,7 @@ export class AgentDeviceSurface { const raw = await this.snapshotOrEmpty(operation, false, budget); const projected = this.project(raw); const found = this.refind(entry, projected.index); - if (input && (isTruncated(raw) || found === undefined || - entry.node.inputPurpose !== found.node.inputPurpose || - entry.node.states?.disabled !== found.node.states?.disabled || - entry.node.states?.secure !== found.node.states?.secure || - entry.node.states?.hidden !== found.node.states?.hidden)) { - throw new EngineError('NODE_STALE', `input node ${entry.id} changed before mutation; observe and authorize it again`, { retryable: true }); - } + if (input) this.refuseChangedInput(entry, found, raw); if (found === undefined) { throw new EngineError('NODE_STALE', `node ${entry.id} changed during transition; re-resolve`, { retryable: true }); } @@ -1421,11 +1562,13 @@ export class AgentDeviceSurface { /** * Records an action the device received: the screen as it was projected - * before it, and when it returned. The transition budget counts from the - * return, since the input lands late in the command's own round trip. + * before it, what kind of action it was, and when it returned. The + * transition budget of that kind counts from the return, since the input + * lands late in the command's own round trip. */ - private markAction(before: ProjectedScreen | undefined): void { + private markAction(before: ProjectedScreen | undefined, kind: ActionKind = 'default'): void { this.screenBeforeAction = before; + this.lastActionKind = kind; this.lastActionAt = Date.now(); } @@ -1454,8 +1597,7 @@ export class AgentDeviceSurface { (client) => input.send(() => client.interactions.scroll({ direction: action.direction, ...settle, ...stabilization })), operation.signal, ); - this.noteGesture(operation); - this.markAction(before); + this.markAction(before, 'gesture'); return; } const entry = this.resolveRef(ref); @@ -1465,6 +1607,7 @@ export class AgentDeviceSurface { if (filling && !entry.takesText) throw refuseFill(entry); const label = `perform ${action.kind}`; const client = this.requireClient(); + const holdMs = action.kind === 'longPress' ? (action.durationMs ?? DEFAULT_LONG_PRESS_MS) : 0; // A toggle already in the wanted state sends nothing, so it neither waits // for a transition nor counts as an action the next control must wait on. if ((action.kind === 'check' || action.kind === 'uncheck') && entry.node.states?.checked === (action.kind === 'check')) { @@ -1474,15 +1617,11 @@ export class AgentDeviceSurface { const run = async (): Promise => { let target = entry; if (filling) { - if (deterministic) await this.waitForTransition(entry, operation, input); - target = await this.relocated(entry, operation, input, true); + const waited = deterministic ? await this.waitForTransition(entry, operation, input, true) : false; + target = typeof waited === 'object' ? waited : await this.relocated(entry, operation, input, true); if (!target.takesText) throw refuseFill(target); } else if (deterministic) { - target = await this.settled(entry, operation, input); - } - if (deterministic) { - target = await this.steadied(target, operation, input, filling); - if (filling && !target.takesText) throw refuseFill(target); + target = await this.settled(entry, operation, input, holdMs); } switch (action.kind) { case 'tap': @@ -1498,8 +1637,8 @@ export class AgentDeviceSurface { return input.press(client, { ...this.actionTarget(target), count: 2, ...settle }); case 'longPress': return input.send( - () => client.interactions.longPress({ ...this.actionTarget(target), ...settle, durationMs: action.durationMs ?? DEFAULT_LONG_PRESS_MS }), - action.durationMs ?? DEFAULT_LONG_PRESS_MS, + () => client.interactions.longPress({ ...this.actionTarget(target), ...settle, durationMs: holdMs }), + holdMs, ); case 'hover': return input.send(() => client.interactions.hover(this.actionTarget(target))); @@ -1528,9 +1667,7 @@ export class AgentDeviceSurface { const rect = target.node.rect; if (rect === undefined) throw notActionable(`node ${target.id} has no bounds to swipe within`); const stabilization = gestureStabilization(operation); - const sent = await input.send(() => client.interactions.swipe({ ...swipeWithin(rect, action.direction, action.momentum), ...stabilization })); - this.noteGesture(operation); - return sent; + return input.send(() => client.interactions.swipe({ ...swipeWithin(rect, action.direction, action.momentum), ...stabilization })); } case 'dragTo': { // A test locates the drop target in a capture of its own, after the @@ -1556,7 +1693,7 @@ export class AgentDeviceSurface { } catch (cause) { throw staleOr(cause, label, this.where); } - this.markAction(before); + this.markAction(before, action.kind === 'swipe' ? 'gesture' : 'default'); } /** @@ -1598,8 +1735,7 @@ export class AgentDeviceSurface { const settle = operation.origin === 'test' ? {} : this.settleOptions; const before = this.latestScreen; await this.command(`${action.kind} at point`, (client) => pointerInteraction(client, point, action, settle, gestureStabilization(operation), input), operation.signal); - if (action.kind === 'swipeTo') this.noteGesture(operation); - this.markAction(before); + this.markAction(before, action.kind === 'swipeTo' ? 'gesture' : 'default'); } /** diff --git a/packages/mobile/tests/types/options.ts b/packages/mobile/tests/types/options.ts index a666c1bab8..2b4f688330 100644 --- a/packages/mobile/tests/types/options.ts +++ b/packages/mobile/tests/types/options.ts @@ -1,11 +1,15 @@ /** Compile-time assertions for the device engine's options and the hosted device provider seam. */ -import type { DeviceLease, DeviceProvider, DeviceRequest, DeviceSession, MobileOptions } from '../../src/index.ts'; +import type { DeviceLease, DeviceProvider, DeviceRequest, DeviceSession, MobileOptions, TransitionBudgets } from '../../src/index.ts'; ({ platform: 'ios', device: 'iPhone 16' }) satisfies MobileOptions; ({ platform: 'android', device: ['Pixel_8', 'Pixel_9'] }) satisfies MobileOptions; ({ platform: 'ios', settle: 200, transition: 500, snapshot: 'interactive' }) satisfies MobileOptions; // `false` skips the settle wait, and a value read from the environment needs no conditional spread. ({ platform: 'ios', settle: false, device: process.env['E2E_DEVICE'] }) satisfies MobileOptions; +// A transition budget per kind of action. Each is optional and takes `undefined`. +const budgets = { default: 750, gesture: 1_500, launch: 500 } satisfies TransitionBudgets; +({ platform: 'ios', transition: budgets }) satisfies MobileOptions; +({ platform: 'android', transition: { gesture: 1_500, launch: undefined } }) satisfies MobileOptions; const farm = { name: 'farm', @@ -25,8 +29,12 @@ const farm = { ({ platform: 'ios', snapshot: 'all' }) satisfies MobileOptions; // @ts-expect-error settle is milliseconds or false; true names no window. ({ platform: 'ios', settle: true }) satisfies MobileOptions; -// @ts-expect-error the transition budget is a number of milliseconds. +// @ts-expect-error the transition budget is a number of milliseconds, or an object of them by kind of action. ({ platform: 'ios', transition: '500ms' }) satisfies MobileOptions; +// @ts-expect-error a budget by kind of action is a number of milliseconds. +({ platform: 'ios', transition: { gesture: '1500ms' } }) satisfies MobileOptions; +// @ts-expect-error the kinds of action are default, gesture, and launch. +({ platform: 'ios', transition: { scroll: 1_500 } }) satisfies MobileOptions; // @ts-expect-error a device is a name, a pool of names, or a provider; a lease is none of those. ({ platform: 'ios', device: { id: 'lease-1', daemon: { baseUrl: 'http://10.0.0.7:4700' } } }) satisfies MobileOptions; // @ts-expect-error a provider releases what it leased. diff --git a/packages/mobile/tests/unit/engine.test.ts b/packages/mobile/tests/unit/engine.test.ts index faf577ac38..82d1961221 100644 --- a/packages/mobile/tests/unit/engine.test.ts +++ b/packages/mobile/tests/unit/engine.test.ts @@ -518,6 +518,22 @@ describe('lifecycle', () => { expect(() => harness({ transition: 0.5 })).toThrow(/non-negative integer/); }); + it.each([ + ['a negative number', -5, 'mobile: `transition` must be a non-negative integer of milliseconds'], + ['a fraction', 0.5, 'mobile: `transition` must be a non-negative integer of milliseconds'], + ['a string', '500ms', 'mobile: `transition` must be a non-negative integer of milliseconds, or an object of them by kind of action (default, gesture, launch)'], + ['an array', [500], 'mobile: `transition` must be a non-negative integer of milliseconds, or an object of them by kind of action (default, gesture, launch)'], + ['a negative default budget', { default: -1 }, 'mobile: `transition.default` must be a non-negative integer of milliseconds'], + ['a fractional gesture budget', { gesture: 1.5 }, 'mobile: `transition.gesture` must be a non-negative integer of milliseconds'], + ['a launch budget that is no number', { launch: '500' }, 'mobile: `transition.launch` must be a non-negative integer of milliseconds'], + ['a misspelled kind of action', { gesure: 1_500 }, 'mobile: `transition` has unknown key "gesure"; did you mean "gesture"?'], + ['a kind of action it does not know', { scroll: 1_500 }, 'mobile: `transition` has unknown key "scroll"; expected one of default, gesture, launch'], + ])('refuses %s as the transition', (_what, transition, message) => { + const refusal = expect.objectContaining({ code: 'INVALID_CONFIG', message }); + expect(() => harness({ transition: transition as never })).toThrowError(refusal); + expect(() => mobile({ platform: 'ios', transition: transition as never })).toThrowError(refusal); + }); + it('rejects a blank device name at config time', () => { expect(() => harness({ device: '' })).toThrow(/must not be blank/); expect(() => harness({ device: ['iPhone 17', ' '] })).toThrow(/must not be blank/); @@ -3540,91 +3556,6 @@ describe('deterministic actions', () => { expect(h.fake.lastArgs('interactions.swipe')).toEqual({ from: expect.any(Object), to: expect.any(Object) }); }); - it('re-reads a control until it rests before a test acts on it after a gesture that skipped the stabilization wait', async () => { - autoAdvanceTimers(); - const h = harness({ settle: false, transition: 0 }); - await openAttempt(h); - await sleep(0); - const root = await screenRootOf(h); - const about = await observed(h, 'About'); - let reads = 0; - h.fake.respond('capture.snapshot', () => { - reads += 1; - const y = reads === 1 ? 150 : 180; - return { ...SETTINGS_SNAPSHOT, nodes: SETTINGS_NODES.map((node) => (node.ref === '@e4' ? { ...node, rect: { ...node.rect!, y } } : node)) }; - }); - - await h.engine.perform!(root.ref, { kind: 'swipe', direction: 'down' }, test()); - await h.engine.perform!(about.ref, { kind: 'tap' }, test()); - // Two reads while the control still moved, one more to see it rest. - expect(reads).toBe(3); - expect(h.fake.methods().slice(-2)).toEqual(['capture.snapshot', 'interactions.press']); - - reads = 10; - await h.engine.perform!(about.ref, { kind: 'tap' }, test()); - expect(reads).toBe(10); - }); - - it('keeps the input checks on every re-read, so a field that turned disabled after a gesture is never filled', async () => { - autoAdvanceTimers(); - const h = harness({ settle: false, transition: 0 }); - await openAttempt(h); - await sleep(0); - const root = await screenRootOf(h); - const search = await observed(h, 'Search'); - let reads = 0; - h.fake.respond('capture.snapshot', () => { - reads += 1; - const field = reads === 1 ? {} : { enabled: false }; - return { ...SETTINGS_SNAPSHOT, nodes: SETTINGS_NODES.map((node) => (node.ref === '@e7' ? { ...node, ...field, rect: { ...node.rect!, y: 220 + reads * 10 } } : node)) }; - }); - - await h.engine.perform!(root.ref, { kind: 'swipe', direction: 'down' }, test()); - await expect(h.engine.perform!(search.ref, { kind: 'fill', value: 'blue', sensitive: false }, test())).rejects.toMatchObject({ - code: 'NODE_STALE', - retryable: true, - }); - expect(h.fake.methods()).not.toContain('interactions.fill'); - }); - - it('stops re-reading a moving control once the action budget cannot fit another poll, and sends nothing', async () => { - autoAdvanceTimers(); - const h = harness({ settle: false, transition: 0 }); - await openAttempt(h); - await sleep(0); - const root = await screenRootOf(h); - const about = await observed(h, 'About'); - let reads = 0; - h.fake.respond('capture.snapshot', () => { - reads += 1; - return { ...SETTINGS_SNAPSHOT, nodes: SETTINGS_NODES.map((node) => (node.ref === '@e4' ? { ...node, rect: { ...node.rect!, y: 120 + reads * 10 } } : node)) }; - }); - - await h.engine.perform!(root.ref, { kind: 'swipe', direction: 'down' }, test()); - await expect(h.engine.perform!(about.ref, { kind: 'tap' }, { ...test(), timeoutMs: 150 })).rejects.toMatchObject({ code: 'OPERATION_TIMEOUT' }); - expect(h.fake.methods()).not.toContain('interactions.press'); - }); - - it('re-reads a control before a node swipe too, so the swipe starts from where the control rests', async () => { - autoAdvanceTimers(); - const h = harness({ settle: false, transition: 0 }); - await openAttempt(h); - await sleep(0); - const root = await screenRootOf(h); - const scroller = await observed(h, 'Scroller'); - let reads = 0; - h.fake.respond('capture.snapshot', () => { - reads += 1; - return { ...SETTINGS_SNAPSHOT, nodes: SETTINGS_NODES.map((node) => (node.ref === '@e10' ? { ...node, rect: { ...node.rect!, y: 350 } } : node)) }; - }); - - await h.engine.perform!(root.ref, { kind: 'swipe', direction: 'down' }, test()); - await h.engine.perform!(scroller.ref, { kind: 'swipe', direction: 'down' }, test()); - // The control moved from y 320 to 350 after the gesture, then rested. - expect(reads).toBe(2); - expect(h.fake.lastArgs('interactions.swipe')).toMatchObject({ from: { y: expect.any(Number) } }); - }); - it('does not re-read a control after a gesture when the agent made it', async () => { const h = harness({ settle: false, transition: 0 }); await openAttempt(h); @@ -3677,5 +3608,491 @@ describe('deterministic actions', () => { })); await tapsAtOnce(h, await observed(h, 'Save'), { ref: '@e11' }); }); + + describe('transition budget by the kind of the last action', () => { + /** Three different budgets, so the wait a step took names the one it used. */ + const BUDGETS = { default: 200, gesture: 800, launch: 50 } as const; + type Kind = keyof typeof BUDGETS; + type Step = (h: Harness) => Promise; + const submit: RawNode = { ref: '@e11', index: 11, parentIndex: 0, depth: 1, type: 'button', label: 'Submit', rect: { x: 0, y: 600, width: 390, height: 44 } }; + + /** + * How long a test's tap on a control that came with `last`, the last thing + * the test did, waits before it is sent. The clock is faked, so this is the + * budget the wait used, not a measurement. The launch's own wait is over + * by the time `last` runs. + */ + async function waitedAfter(h: Harness, last: Step): Promise { + await openAttempt(h); + await sleep(1_000); + await last(h); + const startedAt = Date.now(); + h.fake.respond('capture.snapshot', () => ({ ...SETTINGS_SNAPSHOT, nodes: [...SETTINGS_NODES, submit] })); + await h.engine.perform!((await observed(h, 'Submit')).ref, { kind: 'tap' }, test()); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11' }); + return Date.now() - startedAt; + } + + const relaunch: Step = (h) => h.engine.session!.restart!(test()); + const tap: Step = async (h) => h.engine.perform!((await observed(h, 'About')).ref, { kind: 'tap' }, test()); + const scroll: Step = async (h) => h.engine.perform!((await screenRootOf(h)).ref, { kind: 'swipe', direction: 'down' }, test()); + + const LAST_ACTIONS: [what: string, kind: Kind, last: Step][] = [ + ['an app relaunch', 'launch', relaunch], + ['an app close', 'launch', (h) => h.surface.closeApp(test().signal)], + ['a link opened into the app', 'launch', (h) => h.surface.openLink(new URL('https://example.com/orders/42'), undefined, test().signal)], + ['a tap', 'default', tap], + ['a fill', 'default', async (h) => h.engine.perform!((await observed(h, 'Search')).ref, { kind: 'fill', value: 'blue', sensitive: false }, test())], + [ + 'a drag', + 'default', + async (h) => { + const { root } = await h.engine.observe!(operation()); + return h.engine.perform!(named(root, 'About').ref, { kind: 'dragTo', target: named(root, 'Back').ref }, test()); + }, + ], + ['a tap at a point', 'default', (h) => h.engine.performAt!({ x: 10, y: 20 }, { kind: 'tap' }, test())], + ['a back', 'default', (h) => h.engine.session!.back!(test())], + ['a tap that follows a scroll', 'default', async (h) => { await scroll(h); await tap(h); }], + ['a scroll of the screen', 'gesture', scroll], + ['a swipe on a control', 'gesture', async (h) => h.engine.perform!((await observed(h, 'Scroller')).ref, { kind: 'swipe', direction: 'down' }, test())], + ['a swipe to a point', 'gesture', (h) => h.engine.performAt!({ x: 10, y: 20 }, { kind: 'swipeTo', target: { x: 10, y: 300 } }, test())], + ['a timed swipe to a point', 'gesture', (h) => h.engine.performAt!({ x: 10, y: 20 }, { kind: 'swipeTo', target: { x: 10, y: 300 }, durationMs: 400 }, test())], + ]; + + it.each(LAST_ACTIONS)('after %s the wait is the %s budget', async (_what, kind, last) => { + expect(await waitedAfter(harness({ transition: BUDGETS }), last)).toBe(BUDGETS[kind]); + }); + + it.each(LAST_ACTIONS)('after %s a number is one budget for every kind of action', async (_what, _kind, last) => { + const asNumber = await waitedAfter(harness({ transition: 300 }), last); + const asObject = await waitedAfter(harness({ transition: { default: 300, gesture: 300, launch: 300 } }), last); + expect([asNumber, asObject]).toEqual([300, 300]); + }); + + it.each([ + ['a budget for every kind', BUDGETS, { launch: 50, default: 200, gesture: 800 }], + ['default alone, which gesture and launch then take', { default: 120 }, { launch: 120, default: 120, gesture: 120 }], + ['gesture alone, leaving the others at 500', { gesture: 900 }, { launch: 500, default: 500, gesture: 900 }], + ['launch alone, leaving the others at 500', { launch: 40 }, { launch: 40, default: 500, gesture: 500 }], + ['default and gesture, which launch then follows with default', { default: 150, gesture: 700 }, { launch: 150, default: 150, gesture: 700 }], + ['no budget at all', {}, { launch: 500, default: 500, gesture: 500 }], + ['budgets that are undefined', { default: undefined, gesture: undefined, launch: undefined }, { launch: 500, default: 500, gesture: 500 }], + ['no transition', undefined, { launch: 500, default: 500, gesture: 500 }], + ['a zero gesture budget, which sleeps for nothing', { gesture: 0 }, { launch: 500, default: 500, gesture: 0 }], + ])('waits per kind of action with %s', async (_what, transition, expected) => { + expect({ + launch: await waitedAfter(harness({ transition }), relaunch), + default: await waitedAfter(harness({ transition }), tap), + gesture: await waitedAfter(harness({ transition }), scroll), + }).toEqual(expected); + }); + }); + + describe('a control that came with a gesture waits until it is seen at rest', () => { + type Rows = (read: number, elapsedMs: number) => readonly number[]; + + const textQuery = (value: string): LocatorExpression => ({ kind: 'query', query: { kind: 'text', value: { kind: 'string', value, exact: true } } }); + const presses = (h: Harness): number => h.fake.methods().filter((method) => method === 'interactions.press').length; + + /** Every read past the end of the list shows its last entry. */ + const byRead = + (reads: readonly (readonly number[])[]): Rows => + (read) => + reads[Math.min(read, reads.length - 1)] ?? []; + + /** The screen with a control labelled `label` at each of `ys`; `generation` names the capture, so a ref carries the read it came from. */ + function screenOf(ys: readonly number[], generation: number, label = 'Row 42', type = 'button') { + const rows = ys.map((y, at): RawNode => ({ ref: `@e${11 + at}`, index: 11 + at, parentIndex: 0, depth: 1, type, label, rect: { x: 0, y, width: 390, height: 44 } })); + return { ...SETTINGS_SNAPSHOT, refsGeneration: generation, nodes: [...SETTINGS_NODES, ...rows] }; + } + + /** + * Scrolls the screen as a test does, once the launch's own wait is over, + * and returns when the scroll returned: the moment the budgets count from. + */ + async function scrolled(h: Harness): Promise { + await openAttempt(h); + await sleep(1_000); + await h.engine.perform!((await screenRootOf(h)).ref, { kind: 'swipe', direction: 'down' }, test()); + return Date.now(); + } + + /** + * Answers every snapshot from here on with the controls `rowsOf` gives for + * it, one per y, named `label`. A read takes `latencyMs` on the faked clock + * and shows the screen as of its end. `times` collects when each began and + * `ys` where the first control sat in each, counted from the first read here. + */ + function scrollScreen( + h: Harness, + rowsOf: Rows, + { label = 'Row 42', type = 'button', latencyMs = 0 }: { label?: string; type?: string; latencyMs?: number | ((read: number) => number) } = {}, + ): { times: number[]; ys: (number | undefined)[] } { + const scrolledAt = Date.now(); + const times: number[] = []; + const ys: (number | undefined)[] = []; + h.fake.respond('capture.snapshot', async () => { + const read = times.length; + times.push(Date.now()); + const latency = typeof latencyMs === 'function' ? latencyMs(read) : latencyMs; + if (latency > 0) await sleep(latency); + const rows = rowsOf(read, Date.now() - scrolledAt); + ys.push(rows[0]); + return screenOf(rows, read + 1, label, type); + }); + return { times, ys }; + } + + /** A test's tap on the control the screen now lists under `name`, resolved on a read of its own. */ + async function tapOnScreen(h: Harness, step: OperationContext = test(), name = 'Row 42'): Promise { + await h.engine.perform!((await observed(h, name)).ref, { kind: 'tap' }, step); + } + + it('sleeps the least wait, reads once, and taps where it read the control when it already sits still', async () => { + const h = harness({ transition: { gesture: 800 } }); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, byRead([[500]])); + await tapOnScreen(h); + // The entry's own read, then the one after the least wait that the tap rides on. + expect(times.map((at) => at - t0)).toEqual([0, 800]); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s2' }); + expect(Date.now() - t0).toBe(800); + }); + + it('counts the least wait from the gesture, reads nothing before it is over, and takes a number for it', async () => { + const h = harness({ transition: 1_200 }); + const t0 = await scrolled(h); + await sleep(300); + const { times } = scrollScreen(h, byRead([[500]])); + await tapOnScreen(h); + expect(times.map((at) => at - t0)).toEqual([300, 1_200]); + expect(Date.now() - t0).toBe(1_200); + }); + + it('keeps reading while the control moves and taps it where it came to rest, only after the read that saw it there', async () => { + const h = harness({ transition: { gesture: 800 } }); + const t0 = await scrolled(h); + // The entry's read, four reads of a fling slowing down (the last step is a whole pixel, which is still moving), then one that agrees. + const { times } = scrollScreen(h, byRead([[450], [520], [508], [505], [504], [504]])); + await tapOnScreen(h); + expect(times).toHaveLength(6); + expect(h.fake.methods().slice(-2)).toEqual(['capture.snapshot', 'interactions.press']); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s6' }); + expect(Date.now()).toBe(times[5]); + expect(times[1]! - t0).toBe(800); + const gaps = times.slice(2).map((at, read) => at - times[read + 1]!); + expect(gaps.every((gap) => gap > 0 && gap <= 100)).toBe(true); + }); + + it('goes ahead at the cap with the latest read when the control never stops moving', async () => { + // A least wait that the pauses between reads do not land on the cap from, so the last pause has to be cut short. + const h = harness({ transition: { gesture: 830 } }); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, (read) => [100 + 10 * read]); + await tapOnScreen(h); + expect(Date.now() - t0).toBe(4_000); + expect(times.at(-1)! - t0).toBe(4_000); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: `@e11~s${times.length}` }); + expect(presses(h)).toBe(1); + }); + + it.each([ + ['a tap', 'interactions.press', 'Row 42', 'button', { kind: 'tap' }], + ['a fill', 'interactions.fill', 'Notes', 'text-field', { kind: 'fill', value: 'blue', sensitive: false }], + ] as const)('goes ahead with %s on the latest read, before its deadline, when its timeout cannot fit another read', async (_what, method, label, type, action) => { + // A short least wait and a cap 4 s out, so the action's own timeout is what ends the wait. + const h = harness({ transition: { gesture: 100 } }); + await scrolled(h); + const { times } = scrollScreen(h, (read) => [100 + 10 * read], { label, type }); + const control = await observed(h, label); + const startedAt = Date.now(); + await h.engine.perform!(control.ref, action, { ...test(), timeoutMs: 1_000 }); + expect(h.fake.methods().filter((sent) => sent === method)).toHaveLength(1); + // It reads for rest until another read and the pause before it would not fit, then acts. Nothing starts after the deadline. + expect(times.every((at) => at - startedAt < 1_000)).toBe(true); + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(900); + expect(Date.now() - startedAt).toBeLessThan(1_000); + }); + + it('goes ahead with a long press on the latest read, with its hold still inside the timeout', async () => { + const h = harness({ transition: { gesture: 100 } }); + await scrolled(h); + const { times } = scrollScreen(h, (read) => [100 + 10 * read]); + const control = await observed(h, 'Row 42'); + const startedAt = Date.now(); + await h.engine.perform!(control.ref, { kind: 'longPress', durationMs: 500 }, { ...test(), timeoutMs: 1_000 }); + expect(h.fake.lastArgs('interactions.longPress')).toEqual({ ref: `@e11~s${times.length}`, durationMs: 500 }); + // It stopped reading while 500 ms of the timeout were left for the hold. + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(400); + expect(Date.now() - startedAt).toBeLessThanOrEqual(500); + }); + + it('fails at its deadline with a read that stalls, and sends nothing when that read answers late', async () => { + const h = harness({ transition: { gesture: 100 } }); + await scrolled(h); + // The control's own read and the first read for rest answer at once. The next one outlasts the deadline. + const { times } = scrollScreen(h, (read) => [100 + 10 * read], { latencyMs: (read) => (read < 2 ? 0 : 5_000) }); + const control = await observed(h, 'Row 42'); + const startedAt = Date.now(); + await expect(h.engine.perform!(control.ref, { kind: 'tap' }, { ...test(), timeoutMs: 1_000 })).rejects.toMatchObject({ code: 'OPERATION_TIMEOUT', retryable: false }); + // The stalled read began inside the budget. The call does not wait for the device. + expect(Date.now() - startedAt).toBe(1_000); + const reads = times.length; + await sleep(6_000); + expect(times).toHaveLength(reads); + expect(presses(h)).toBe(0); + }); + + it('takes the one read after a gesture budget above the cap, which is then the cap, and goes ahead with it', async () => { + const h = harness({ transition: { gesture: 5_000 } }); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, (read) => [100 + 10 * read]); + await tapOnScreen(h); + expect(times.map((at) => at - t0)).toEqual([0, 5_000]); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s2' }); + }); + + it('gives a gesture budget above the cap its whole sleep, counted from the gesture, and one read after it', async () => { + const h = harness({ transition: { gesture: 5_000 } }); + const t0 = await scrolled(h); + await sleep(4_500); + const { times } = scrollScreen(h, byRead([[500]])); + await tapOnScreen(h); + expect(times.map((at) => at - t0)).toEqual([4_500, 5_000]); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s2' }); + }); + + it.each([ + ['a gesture budget of 0', { transition: { gesture: 0 } }], + ['a transition of 0', { transition: 0 }], + ] as const)('goes ahead at once with %s, with no sleep and no read for rest', async (_what, options) => { + const h = harness(options); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, byRead([[450], [480], [480]])); + await tapOnScreen(h); + // Only the read the tap resolved on. + expect(times).toHaveLength(1); + expect(Date.now()).toBe(t0); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s1' }); + }); + + it('still looks for rest when the least wait is a single millisecond', async () => { + const h = harness({ transition: { gesture: 1 } }); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, byRead([[450], [480], [480]])); + await tapOnScreen(h); + expect(times).toHaveLength(3); + expect(times[1]! - t0).toBe(1); + expect(times[2]! - times[1]!).toBeLessThanOrEqual(100); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s3' }); + }); + + it.each([ + ['the read the action resolved on', [[500, 600], [400, 500], [300, 400], [300, 400]]], + ['the read before it', [[300, 400], [500, 600], [400, 500], [400, 500]]], + ])('does not take a repeated row for at rest when the list moved a whole row since %s', async (_what, reads) => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead(reads), { label: 'Add' }); + await tapOnScreen(h, test(), 'Add'); + // A read that matched the control alone would call the row now sitting where it was "still" at the second read. + expect(times).toHaveLength(4); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e12~s4' }); + }); + + it.each([ + ['the newest read', false, 2], + ['a read that a newer one has replaced', true, 4], + ])('knows the look-alikes of a located control when its read is %s', async (_what, replaced, reads) => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead([[500, 600]]), { label: 'Add' }); + const [add] = await h.engine.locate!(textQuery('Add'), test()); + if (replaced) await h.engine.observe!(operation()); + await h.engine.perform!(add!.ref, { kind: 'tap' }, test()); + // Without its own read the control stands alone, which two look-alikes do not match: one more read has to agree with the one before. + expect(times).toHaveLength(reads); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: `@e11~s${reads}` }); + }); + + it.each([ + ['at the first read after the least wait', [[500], []], 2], + ['while it is still moving', [[500], [480], [460], []], 4], + ])('fails with a retryable NODE_STALE, with no tap and no wait for the cap, when the control is gone %s', async (_when, reads, taken) => { + const h = harness({ transition: { gesture: 800 } }); + const t0 = await scrolled(h); + const { times } = scrollScreen(h, byRead(reads)); + await expect(tapOnScreen(h)).rejects.toMatchObject({ code: 'NODE_STALE', retryable: true }); + expect(times).toHaveLength(taken); + expect(presses(h)).toBe(0); + expect(Date.now() - t0).toBeLessThan(4_000); + }); + + it.each([ + ['the least wait', 100, [[500]], 1], + ['the pause between two reads', 820, [[500], [480], [480]], 2], + ])('is cancelled by an abort during %s, with no read after it and no tap', async (_when, abortAt, reads, taken) => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead(reads)); + const controller = new AbortController(); + const pending = tapOnScreen(h, { ...operation(controller.signal), origin: 'test' }); + setTimeout(() => controller.abort(), abortAt); + await expect(pending).rejects.toMatchObject({ code: 'CANCELLED' }); + expect(times).toHaveLength(taken); + expect(presses(h)).toBe(0); + }); + + it('is cancelled by an abort during a read, with no tap', async () => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + scrollScreen(h, byRead([[500]])); + const control = await observed(h, 'Row 42'); + const controller = new AbortController(); + h.fake.respond('capture.snapshot', async () => { + controller.abort(); + return screenOf([480], 2); + }); + await expect(h.engine.perform!(control.ref, { kind: 'tap' }, { ...operation(controller.signal), origin: 'test' })).rejects.toMatchObject({ code: 'CANCELLED' }); + expect(presses(h)).toBe(0); + }); + + it.each([ + [300, 'reads once for rest', 2], + [3_999, 'reads once for rest', 2], + [4_000, 'reads nothing, the cap being over', 1], + ])('sleeps no more once the gesture returned %s ms ago, and %s', async (ago, _what, taken) => { + const h = harness({ transition: { gesture: 100 } }); + await scrolled(h); + await sleep(ago); + const { times } = scrollScreen(h, byRead([[500]])); + const control = await observed(h, 'Row 42'); + await tapsAtOnce(h, control, { ref: `@e11~s${taken}` }); + expect(times).toHaveLength(taken); + }); + + it.each([ + ['a tap', async (h: Harness) => h.engine.perform!((await observed(h, 'About')).ref, { kind: 'tap' }, test())], + ['a relaunch', (h: Harness) => h.engine.session!.restart!(test())], + ])('does not look for rest after %s: one read after its budget, however the control moves', async (_what, last) => { + const h = harness({ transition: 300 }); + await openAttempt(h); + await sleep(1_000); + await last(h); + const t0 = Date.now(); + const { times } = scrollScreen(h, (read) => [100 + 10 * read]); + await tapOnScreen(h); + expect(times.map((at) => at - t0)).toEqual([0, 300]); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s2' }); + }); + + it('waits out a fling that outlasts the budget and taps the row where the list stopped', async () => { + const h = harness({ transition: { gesture: 1_500 } }); + const t0 = await scrolled(h); + // Modelled on an iOS simulator, where 3 of 120 scrolls of a long list left it moving 1.66 to 1.84 s after `scroll` returned. A read takes 170 ms here. + const stoppedAt = 1_840; + const rowAt = (elapsedMs: number): number => (elapsedMs >= stoppedAt ? 504 : 504 + 300 * (1 - elapsedMs / stoppedAt) ** 2); + const { times, ys } = scrollScreen(h, (_read, elapsedMs) => [rowAt(elapsedMs)], { latencyMs: 170 }); + await tapOnScreen(h); + expect(ys.at(-1)).toBe(504); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: `@e11~s${times.length}` }); + expect(Date.now() - t0).toBeGreaterThanOrEqual(stoppedAt); + }); + + it('fills a field that came with a gesture on the read that found it at rest, with no read of its own', async () => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead([[450], [500], [504], [504]]), { label: 'Notes', type: 'text-field' }); + const notes = await observed(h, 'Notes'); + await h.engine.perform!(notes.ref, { kind: 'fill', value: 'blue', sensitive: false }, test()); + // The entry's read and three reads for rest. The last of them is the re-read before the fill types. + expect(times).toHaveLength(4); + expect(h.fake.lastArgs('interactions.fill')).toEqual({ ref: '@e11~s4', text: 'blue' }); + }); + + it('fills on the latest read, with no read of its own, when its timeout cannot fit another', async () => { + const h = harness({ transition: { gesture: 100 } }); + await scrolled(h); + const { times } = scrollScreen(h, (read) => [100 + 10 * read], { label: 'Notes', type: 'text-field', latencyMs: 250 }); + const notes = await observed(h, 'Notes'); + const startedAt = Date.now(); + await h.engine.perform!(notes.ref, { kind: 'fill', value: 'blue', sensitive: false }, { ...test(), timeoutMs: 1_000 }); + expect(h.fake.lastArgs('interactions.fill')).toEqual({ ref: `@e11~s${times.length}`, text: 'blue' }); + expect(Date.now() - startedAt).toBeLessThan(1_000); + }); + + it.each([ + ['truncated', (screen: ReturnType) => ({ ...screen, truncated: true })], + ['disabled', (screen: ReturnType) => ({ ...screen, nodes: screen.nodes.map((node) => (node.label === 'Notes' ? { ...node, enabled: false } : node)) })], + ])('refuses a fill when the read that found the field at rest is %s, and sends nothing', async (_what, change) => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + let reads = 0; + h.fake.respond('capture.snapshot', () => { + const read = reads; + reads += 1; + const screen = screenOf([500], read + 1, 'Notes', 'text-field'); + return read === 0 ? screen : change(screen); + }); + const notes = await observed(h, 'Notes'); + await expect(h.engine.perform!(notes.ref, { kind: 'fill', value: 'blue', sensitive: false }, test())).rejects.toMatchObject({ code: 'NODE_STALE', retryable: true }); + expect(h.fake.methods()).not.toContain('interactions.fill'); + }); + + it('keeps the input checks on the read that finds a field at rest, so a field that turned disabled after a gesture is never filled', async () => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + let read = 0; + h.fake.respond('capture.snapshot', () => { + read += 1; + const screen = screenOf(read === 1 ? [450] : [504], read, 'Notes', 'text-field'); + return read === 1 ? screen : { ...screen, nodes: screen.nodes.map((node) => (node.ref === '@e11' ? { ...node, enabled: false } : node)) }; + }); + const notes = await observed(h, 'Notes'); + await expect(h.engine.perform!(notes.ref, { kind: 'fill', value: 'blue', sensitive: false }, test())).rejects.toMatchObject({ code: 'NODE_STALE', retryable: true }); + expect(h.fake.methods()).not.toContain('interactions.fill'); + }); + + it('reads a control until it rests before a node swipe too, so the swipe starts where the control stopped', async () => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead([[450], [500], [504], [504]])); + await h.engine.perform!((await observed(h, 'Row 42')).ref, { kind: 'swipe', direction: 'down' }, test()); + expect(times).toHaveLength(4); + const from = (h.fake.lastArgs('interactions.swipe') as { from: { y: number } }).from; + expect(from.y).toBeGreaterThanOrEqual(504); + expect(from.y).toBeLessThanOrEqual(548); + }); + + it('acts at once on a control that sits where it did before the gesture', async () => { + const h = harness({ transition: { gesture: 800 } }); + await openAttempt(h); + await sleep(1_000); + await observed(h, 'About'); + await h.engine.perform!((await screenRootOf(h)).ref, { kind: 'swipe', direction: 'down' }, test()); + const looks = h.fake.methods().filter((method) => method === 'capture.snapshot').length; + await tapsAtOnce(h, await observed(h, 'About'), { ref: '@e4' }); + expect(h.fake.methods().filter((method) => method === 'capture.snapshot')).toHaveLength(looks + 1); + }); + + it('does not make an agent action wait for rest', async () => { + const h = harness({ transition: { gesture: 800 } }); + await scrolled(h); + const { times } = scrollScreen(h, byRead([[500]])); + const control = await observed(h, 'Row 42'); + vi.setTimerTickMode('manual'); + try { + const pending = h.engine.perform!(control.ref, { kind: 'tap' }, operation()); + await new Promise((resolve) => setImmediate(resolve)); + expect(vi.getTimerCount()).toBe(0); + expect(h.fake.lastArgs('interactions.press')).toEqual({ ref: '@e11~s1', settle: true, settleQuietMs: 150 }); + await pending; + } finally { + vi.setTimerTickMode('nextTimerAsync'); + } + expect(times).toHaveLength(1); + }); + }); }); diff --git a/plugins/e2e/skills/e2e/references/setup.md b/plugins/e2e/skills/e2e/references/setup.md index 0afc20fb9a..9af6e05195 100644 --- a/plugins/e2e/skills/e2e/references/setup.md +++ b/plugins/e2e/skills/e2e/references/setup.md @@ -406,10 +406,18 @@ export default { `app.appPath` before tests. Set `E2B_API_KEY`; install the `e2b` peer. The runner needs no `adb`. - A control that appeared or moved with the previous action waits out - `transition` (default 500 ms). Opening or closing the keyboard also uses - the remaining budget. Screen swipes in tests wait out the remaining budget - before scrolling. Agent actions settle `settle` ms (default 150) before the - next observation, `settle: false` skips it. + `transition` (default 500 ms). Pass an object to set the wait by the + previous action: `gesture` after a scroll or swipe, `launch` after the app + launches, relaunches, or closes or a link opens into it, and `default` for + the rest. A `gesture` or `launch` left out takes `default`. The `gesture` + budget is the least wait. + The control is then read until it stops moving, which is two reads that + agree, at most 4 s after the gesture, or at the budget when that is larger, + or sooner when the action's own timeout cannot fit another read. A plain + number sets it too, and 0 is no wait at all. Opening or closing the keyboard + also uses the remaining budget. Screen swipes in tests wait out the remaining + budget before scrolling. Agent actions settle `settle` ms (default 150) before + the next observation, `settle: false` skips it. - Mobile `fill` and `clear` re-read the field before typing, replay included. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation. diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index 0afc20fb9a..9af6e05195 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -406,10 +406,18 @@ export default { `app.appPath` before tests. Set `E2B_API_KEY`; install the `e2b` peer. The runner needs no `adb`. - A control that appeared or moved with the previous action waits out - `transition` (default 500 ms). Opening or closing the keyboard also uses - the remaining budget. Screen swipes in tests wait out the remaining budget - before scrolling. Agent actions settle `settle` ms (default 150) before the - next observation, `settle: false` skips it. + `transition` (default 500 ms). Pass an object to set the wait by the + previous action: `gesture` after a scroll or swipe, `launch` after the app + launches, relaunches, or closes or a link opens into it, and `default` for + the rest. A `gesture` or `launch` left out takes `default`. The `gesture` + budget is the least wait. + The control is then read until it stops moving, which is two reads that + agree, at most 4 s after the gesture, or at the budget when that is larger, + or sooner when the action's own timeout cannot fit another read. A plain + number sets it too, and 0 is no wait at all. Opening or closing the keyboard + also uses the remaining budget. Screen swipes in tests wait out the remaining + budget before scrolling. Agent actions settle `settle` ms (default 150) before + the next observation, `settle: false` skips it. - Mobile `fill` and `clear` re-read the field before typing, replay included. Missing fields, changed security state, or incomplete captures return `NODE_STALE` before mutation, then retry after a fresh observation.