Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/mobile-gesture-rest-wait.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@e2e-dev/mobile": patch
---

`mobile({ transition })` takes an object, `{ default, gesture, launch }` (the new `TransitionBudgets` type), to set the wait before acting on a control by the kind of the last action. A number still sets the same wait after every action. After a scroll or swipe, a test now waits for a control that came with it to stop moving: the `gesture` budget is the least wait, then the control is read until two reads agree. The wait ends at the latest 4 s after the gesture (the `gesture` budget when that is larger), or sooner when the action's timeout cannot fit another read, and the action then goes ahead on the latest read. A budget of 0 is still no wait at all.
7 changes: 4 additions & 3 deletions apps/mobile-benchmark/e2e.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,10 @@ export const ios = mobile({
session: 'e2e-mobile-benchmark-ios',
// A control that just moved waits this long to come to rest before a tap;
// an unmoved one is tapped at once, so the budget costs only after a
// scroll. iOS reports a row's final frame from the first frame of a fling,
// and the default half second let a tap after scrollUntilVisible land on
// the row still passing underneath on a loaded simulator.
// scroll. The default half second let a tap after scrollUntilVisible land
// on the row still passing underneath on a loaded simulator. After a scroll
// the engine also reads until the control stops moving, so this budget is
// the least wait.
transition: 1_500,
});

Expand Down
98 changes: 81 additions & 17 deletions docs/reference/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ The app under test is the target's `app`, not an engine option; see
| [`session`](#session) | `string` | `e2e-<target name>` |
| [`snapshot`](#snapshot) | `'full' \| 'interactive'` | `'full'` |
| [`settle`](#settle) | `number \| false` | `150` |
| [`transition`](#transition) | `number` | `500` |
| [`transition`](#transition) | `number \| TransitionBudgets` | `500` |
| [`videoTouches`](#videotouches) | `boolean` | `true` |

Every optional value also accepts `undefined`, so values read straight from
Expand All @@ -99,7 +99,8 @@ declaration.
| `INVALID_CONFIG` | `platform` is missing or not `'ios'` or `'android'`; the match is case-sensitive. |
| `INVALID_CONFIG` | A provided `snapshot` value is not `'full'` or `'interactive'`, or a provided `videoTouches` value is not a boolean. |
| `INVALID_CONFIG` | An option `mobile()` does not take, such as `mobile({ app })` and the other old app options. The message names the nearest known option for a likely typo, else every option. |
| `INVALID_CONFIG` | `settle` or `transition` is not a non-negative integer of milliseconds (`settle` may also be `false`). |
| `INVALID_CONFIG` | `settle` or `transition` is not a non-negative integer of milliseconds. `settle` may also be `false`, and `transition` an object of budgets. |
| `INVALID_CONFIG` | A budget in a `transition` object is not a non-negative integer of milliseconds, or the object has a key other than `default`, `gesture`, and `launch`. The message names the budget, or the nearest known key for a likely typo, else every key. |

#### `platform`

Expand Down Expand Up @@ -232,31 +233,74 @@ mobile({ platform: 'ios', settle: 300 }); // default: 150

```ts
/**
* Wait before acting on a control that just appeared or moved, in
* milliseconds. Default 500. Accessibility bounds cannot tell a sliding
* control from a landed one (iOS reports the final frame at once, Android
* the frame in flight), so the control is found again in a fresh snapshot
* once the wait is over and acted on where it landed. Unmoved controls are
* acted on immediately only when keyboard visibility is unchanged. Opening
* or closing the keyboard also uses the remaining budget. Screen swipes in
* tests wait out the remaining budget before scrolling. Raise it for slower
* transitions.
* How long a control that appeared or moved with the last action is given
* to finish arriving before a test acts on it, in milliseconds. Default
* 500, the length of a modal or screen transition. Accessibility bounds
* cannot tell a sliding control from a landed one (iOS reports the final
* frame at once, Android the frame in flight), so the control is found
* again in a fresh snapshot once the wait is over and acted on where it
* landed. Unmoved controls are acted on immediately only when keyboard
* visibility is unchanged. Opening or closing the keyboard also uses the
* remaining budget. Screen swipes in tests wait out the remaining budget
* before scrolling. Raise it for slower transitions. A number is that
* budget after every action. An object sets it by the kind of the last
* action: `default` for any action except a gesture or a launch, `gesture`
* after a scroll or swipe, `launch` after the app launches, relaunches, or
* closes, or a link opens into it. A `gesture` or `launch` budget left out
* takes the `default` one. A scroll or swipe can return while its content
* still moves, and the tree shows that content in flight, so the `gesture`
* budget is only the least wait. A control that arrived or moved with the
* gesture is then read until it stops moving, which is two reads that
* agree on where it and any control that looks like it sit, at most 4 s
* after the gesture, or at the budget when that is larger, or sooner when
* the action's own timeout cannot fit another read. A plain number sets the
* `gesture` budget too. A budget of 0 is no wait at all. After any other
* action the budget is the whole wait.
*/
readonly transition?: number | undefined;
readonly transition?: number | TransitionBudgets | undefined;
```

```ts
mobile({ platform: 'ios', transition: 800 }); // default: 500
```
mobile({ platform: 'ios', transition: { default: 750, gesture: 1_500, launch: 500 } });
```

An object sets the wait by what the last action was. `gesture` is the wait
after a scroll or swipe, `launch` after the app launches, relaunches, or
closes, or a link opens into it, and `default` after any other action.
`default` is 500 unless set, and `gesture` and `launch` take the `default`
budget when left out, so `{ gesture: 1_500 }` waits 1.5 s after a scroll or
swipe and 500 ms after everything else. Every budget is a non-negative integer
of milliseconds, and a key other than these three is `INVALID_CONFIG`.

The `gesture` budget is the least wait, not the whole of it. A scroll or swipe
can return while its content is still moving, and the tree shows that content
in flight, so the wait goes on until the control stops. After any other action
the budget is the whole wait, since the tree cannot tell a control that slides
in from one that landed. A plain number sets the `gesture` budget too. A
budget of 0 is no wait at all, so `transition: 0` and `{ gesture: 0 }` act at
once.

After the `gesture` budget, the engine reads the screen until a read agrees
with the one before it, or with the one that found the control. Two reads agree
when the control and its look-alikes are the same in number and sit at the same
places. A look-alike has the same role, test ID, name, and text. The wait ends
at most 4 s after the gesture, or at the budget when that is larger, and the
action then goes ahead where the control was last read. A control that never
stops moving costs the cap, 4 s or the `gesture` budget when that is larger, or
what the action's own timeout leaves: the wait ends once the timeout cannot fit
another read, and the action goes ahead where the control was last read.

If the control changes during the wait, the action finds it again within its timeout before acting.

`back`, `home`, `alert`, `dismissKeyboard`, `setOrientation`, and `fold` on
[`device`](#device) change the screen, so they count as actions for this
budget the way a tap does. A control that arrives with the screen they reveal
waits out the budget before the next action lands on it.
budget the way a tap does, and the `default` budget applies after them. A
control that arrives with the screen they reveal waits out the budget before
the next action lands on it.

`fill` and `clear` re-read the field before typing. Missing fields, changed
`fill` and `clear` re-read the field before typing; after a scroll or swipe, the
read that found the field at rest is that re-read. Missing fields, changed
security state, or incomplete captures return `NODE_STALE` before mutation,
then retry after a fresh observation. Non-input targets return
`UNSUPPORTED_CAPABILITY`. An iOS `other`, the type of an element the platform
Expand Down Expand Up @@ -1190,6 +1234,26 @@ export type PermissionState = 'grant' | 'deny' | 'reset';

/** Permissions an app holds when it launches, by name. */
export type LaunchPermissions = Readonly<Partial<Record<DevicePermission, PermissionState>>>;

/**
* The `transition` budgets by the kind of the last action, in milliseconds,
* each a non-negative integer. A `gesture` or `launch` budget left out takes
* the `default` one.
*/
export interface TransitionBudgets {
/** After any action except a gesture or a launch: a tap, fill, key, back, and the rest. Default 500. */
readonly default?: number | undefined;
/**
* After a scroll or swipe, which can return while the content is still
* moving. The least wait. The wait then goes on until the control stops
* moving, two reads agreeing, at most 4 s after the gesture. A budget above
* 4 s ends the wait at that budget, and 0 is no wait at all. Default: the
* `default` budget.
*/
readonly gesture?: number | undefined;
/** After the app launches, relaunches, or closes, or a link opens into it. Default: the `default` budget. */
readonly launch?: number | undefined;
}
```

### Device fixture
Expand Down Expand Up @@ -1289,7 +1353,7 @@ export interface DeviceSelection {
### Exported types

`MobileOptions`, `MobilePlatform`, `DevicePermission`, `LaunchPermissions`,
`PermissionState`, `DeviceClientConfig`, `DeviceConnection`, `DeviceDaemon`,
`PermissionState`, `TransitionBudgets`, `DeviceClientConfig`, `DeviceConnection`, `DeviceDaemon`,
`DeviceProvider`, `DeviceLease`, `DeviceReleaseContext`, `DeviceRequest`,
`Device`, `DeviceSession`, `DeviceSessionClient`, `DeviceSelection`, `DeviceOrientation`, `BiometricSensor`, `ForegroundApp`,
`FoldPose`, `InstallAppOptions`, `InstalledApp`, and `OpenAppOptions` are exported from
Expand Down
2 changes: 1 addition & 1 deletion packages/mobile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ the other old app options are `INVALID_CONFIG` naming their key under `app`.
| `session` | agent-device session name, before the worker slot: slot `n` drives its device under `<session>-<n>`, `e2e-<target name>-<n>` by default. One run per session at a time. |
| `snapshot` | `'full'` (default, includes static text) or `'interactive'` (actionable nodes only). |
| `settle` | Milliseconds the UI must hold still after an agent action before the agent observes again, default `150`; `false` skips the wait. A test's steps do not settle, except `device.back()`; `expect` verifies their outcome. |
| `transition` | For a test's steps: milliseconds a control that appeared or moved with the last action gets to finish arriving before it is acted on, default `500`. Unmoved controls are acted on immediately only when keyboard visibility is unchanged. Opening or closing the keyboard also uses the remaining budget. Screen swipes in tests wait out the remaining budget before scrolling. |
| `transition` | For a test's steps: milliseconds a control that appeared or moved with the last action gets to finish arriving before it is acted on, default `500`. An object sets it by the kind of the last action: `default` for any action except a gesture or a launch, `gesture` after a scroll or swipe, `launch` after the app launches, relaunches, or closes, or a link opens into it. A `gesture` or `launch` budget left out takes `default`. The `gesture` budget is the least wait. The control is then read until it stops moving, which is two reads that agree, at most 4 s after the gesture, or at the budget when that is larger, or sooner when the action's own timeout cannot fit another read. A plain number sets it too, and 0 is no wait at all. Unmoved controls are acted on immediately only when keyboard visibility is unchanged. Opening or closing the keyboard also uses the remaining budget. Screen swipes in tests wait out the remaining budget before scrolling. |
| `videoTouches` | For recordings the mobile engine makes (not a device provider's own recording), whether the video draws agent-device's touch indicator where a test tapped, default `true`. `false` records the plain screen and skips drawing the indicator after the recording stops, which on a hosted daemon can outlast `cleanupTimeout`. |

Every optional `mobile()` value also accepts `undefined`, so a config passes
Expand Down
2 changes: 1 addition & 1 deletion packages/mobile/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import type { Device } from './device.ts';
export { mobile } from './engine.ts';
/** The engine factory again, as the default export: what `e2e mcp` calls when `open_session` names this package as its `engine`. */
export { mobile as default } from './engine.ts';
export type { DevicePermission, LaunchPermissions, MobileOptions, MobilePlatform, PermissionState } from './options.ts';
export type { DevicePermission, LaunchPermissions, MobileOptions, MobilePlatform, PermissionState, TransitionBudgets } from './options.ts';
export type { DeviceClientConfig, DeviceConnection, DeviceDaemon } from './bindings.ts';
export type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from './provider.ts';
export type { FoldPose, InstallAppOptions, InstalledApp, OpenAppOptions } from './surface.ts';
Expand Down
50 changes: 42 additions & 8 deletions packages/mobile/src/options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,26 @@ export type PermissionState = 'grant' | 'deny' | 'reset';
/** Permissions an app holds when it launches, by name. */
export type LaunchPermissions = Readonly<Partial<Record<DevicePermission, PermissionState>>>;

/**
* The `transition` budgets by the kind of the last action, in milliseconds,
* each a non-negative integer. A `gesture` or `launch` budget left out takes
* the `default` one.
*/
export interface TransitionBudgets {
/** After any action except a gesture or a launch: a tap, fill, key, back, and the rest. Default 500. */
readonly default?: number | undefined;
/**
* After a scroll or swipe, which can return while the content is still
* moving. The least wait. The wait then goes on until the control stops
* moving, two reads agreeing, at most 4 s after the gesture. A budget above
* 4 s ends the wait at that budget, and 0 is no wait at all. Default: the
* `default` budget.
*/
readonly gesture?: number | undefined;
/** After the app launches, relaunches, or closes, or a link opens into it. Default: the `default` budget. */
readonly launch?: number | undefined;
}

/**
* Options of the device engine: which devices it drives and how. The app
* under test is the target's `app` (`bundleId`, `appPath`,
Expand Down Expand Up @@ -107,15 +127,29 @@ export interface MobileOptions {
/**
* How long a control that appeared or moved with the last action is given
* to finish arriving before a test acts on it, in milliseconds. Default
* 500, the length of a modal or screen transition. Accessibility frames
* report a control's final position from the first frame of a transition,
* so this budget is the only thing that keeps a test from tapping a point
* the control has not reached. Unmoved controls are acted on immediately
* only when keyboard visibility is unchanged. Opening or closing the
* keyboard also uses the remaining budget. Screen swipes in tests wait out
* the remaining budget before scrolling. Raise it for slower transitions.
* 500, the length of a modal or screen transition. Accessibility bounds
* cannot tell a sliding control from a landed one (iOS reports the final
* frame at once, Android the frame in flight), so the control is found
* again in a fresh snapshot once the wait is over and acted on where it
* landed. Unmoved controls are acted on immediately only when keyboard
* visibility is unchanged. Opening or closing the keyboard also uses the
* remaining budget. Screen swipes in tests wait out the remaining budget
* before scrolling. Raise it for slower transitions. A number is that
* budget after every action. An object sets it by the kind of the last
* action: `default` for any action except a gesture or a launch, `gesture`
* after a scroll or swipe, `launch` after the app launches, relaunches, or
* closes, or a link opens into it. A `gesture` or `launch` budget left out
* takes the `default` one. A scroll or swipe can return while its content
* still moves, and the tree shows that content in flight, so the `gesture`
* budget is only the least wait. A control that arrived or moved with the
* gesture is then read until it stops moving, which is two reads that
* agree on where it and any control that looks like it sit, at most 4 s
* after the gesture, or at the budget when that is larger, or sooner when
* the action's own timeout cannot fit another read. A plain number sets the
* `gesture` budget too. A budget of 0 is no wait at all. After any other
* action the budget is the whole wait.
*/
readonly transition?: number | undefined;
readonly transition?: number | TransitionBudgets | undefined;
/**
* Whether a video recording draws agent-device's touch indicator where a
* test tapped. Default `true`. `false` records the plain screen, and skips
Expand Down
Loading
Loading