Skip to content
Merged
1 change: 1 addition & 0 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ The port installs the skill with `npx skills add hardkoded/e2e-dotnet` and regis
- Upstream's `device.unstable_raw` (`feat(mobile): device.unstable_raw exposes the test's agent-device session`, `c3c4238`, tester-army/e2e#1380) is `Device.UnstableRaw`. Upstream's `DeviceSession` interface (`device.ts`) is `RawDeviceSession`, a .NET-only name, because the port's `DeviceSession` is the surface (upstream's `AgentDeviceSurface`) and a public type of that name exists. Its `client` is the session's `IDeviceClient` (the port's stand-in for `AgentDeviceClient`), `selection` is `DeviceSelection`, and `platform` is `MobilePlatform`. `BoundDevice` (`surface.ts`) is internal, set by the boot from the `id` and `device` of the `devices.boot` answer (`DeviceSession.BoundDevice`, upstream's `boundDevice`), and cleared when the session closes. Reading `UnstableRaw`, or `Client` of a handle taken earlier, after the session closed is `INVALID_STATE`. The client is `DeviceCommands.Translate`, which refuses a method it does not list, so `DeviceMethods.AppsList` (`apps.list`, the daemon's `apps` command) is added for upstream's reference example; the other methods of upstream's client stay unsupported until code calls them. Tests: the three `it` of `describe('unstable_raw')` in `Engine/DeviceFixtureUnstableRawTests`; the first is `Partial_`, since the port's fixture has no map of declared operations for `expect(declared).not.toHaveProperty('unstable_raw')`. Not ported: `unstable_raw reaches the test session through agent-device` (`apps/mobile-benchmark/tests/device-fixture.e2e.ts`), which needs the mobile benchmark app, a React Native app the port lacks.
- Upstream's known-viewport fix (`fix(mobile): keep the known viewport when the app reads smaller than the screen`, `7674a2c`, tester-army/e2e#1291) is ported. `DeviceSession.Project` learns a snapshot's viewport only when it is plainly the screen's: the first one the session reads, the one it knows, or that one rotated (`SameScreen`, upstream's `sameScreen`; two sizes within a point are one screen, since rects from the tree carry fractions). `ViewportForAsync` (upstream's `viewportFor`) checks a tree of any other size against the device's screenshot, in either orientation, before it replaces the known one: a screenshot reports the portrait size even in landscape, so it confirms a size but cannot pick the orientation. The size it checks against is `DeviceSupport.LogicalScreenSize` (upstream's `logicalScreenSize`): the screenshot's logical dimensions, else its pixel dimensions over its density. A simulator's screenshot reports those; a real device's reports neither, so there the tree's size is taken as it comes. A screenshot that fails during that check fails the observation, as upstream's does. `TraceScreen` and `MaskedScreenshotAsync` size the trace screen and the secure-field masks by the known viewport, not the tree's. On iOS, a swipe from a sheet's grabber can pull down Notification Center, and the app's tree behind it reported a third of the screen (130x281 on a 390x844 device), which misplaced every rect and tap after it. Upstream records this only in its changeset; its skill references and docs say nothing about it, so `SKILL.md` and `docs/` have nothing to mirror. Tests: in `Engine/ObservationTests`, 'keeps the known viewport for an app drawn smaller than the screen, and follows a rotation' (new) and 'measures the screen in the viewport the snapshot publishes, and in its tree when it publishes none' (split into two harnesses, as upstream's); in `Engine/DeviceFixtureDismissingAKeyboardThatHasNoDismissKeyTests`, 'drags at the centre of a screen rotated since the last observation' (new), since the keyboard drag measures its start through `ViewportForAsync` too.
- Upstream's input budget (`fix(mobile): bound each input by its budget`, `14b77d3`, tester-army/e2e#1073) is ported. An input (a tap or other node action, a point action, typing, a key, `back`, a keyboard dismissal, and the node `LongPressAsync`, `HoverAsync`, `DragToAsync`, and `SwipeAsync` the port has in place of upstream's `longPress`, `hover`, `dragTo`, and `swipe` actions) runs its device commands under `InputBudget` (`DeviceSupport.cs`, upstream's `support.ts`): with no budget left the input is not sent and fails with `OPERATION_TIMEOUT`, an input that does not answer in time fails with `ACTION_MAY_HAVE_COMMITTED`, one sent with almost no budget left still gets 250 ms to answer, a hold (a long press or timed swipe) at least as long as the budget left is refused with `NOT_ACTIONABLE`, and each command of a multi-command input (a key on an unfocused field) is sent on its own. Reads, launches, artifacts, and the `Device` fixture calls (including `device.dismissKeyboard`, which passes no budget, as upstream's `dismissKeyboard(signal)` does) are unbounded. Differences: upstream reads the budget per call, from `operation.timeoutMs`, which is the call's `timeout` or `actionTimeout`. The port's session calls carry no timeout, so every input gets `DeviceSession.ActionTimeout`, which `MobileStartInfo.ActionTimeout` sets (`EngineStartOptions.ActionTimeout` fills it), as the web engine's operations get the action timeout. `DeviceSession.InputBudget` is upstream's `inputBudget` and `DeviceSession.SendInputAsync` its `sendInput`. `InputBudget`'s `track` is an `Action<Task>` that registers the task (upstream's returns the promise it is given); see [In-flight commands](#in-flight-commands). The `OperationContext.signal` doc change in `docs/reference/engine.mdx` and `contract.ts` has no counterpart: the port has no `OperationContext`. Tests: in `Engine/ObservationTests`, 'ends %s, hung on %s, at its budget as possibly committed' (a theory of the same seven entries; the screen swipe is a `perform` on the root), 'sends nothing once a settling target spent the budget, and no keystroke once a focus tap used it up', and 'refuses a long press that holds longer than its timeout leaves, before sending it' (also on the node `LongPressAsync`); in `Support/SupportHelpersTests`, 'sends an input with what is left of its budget, and reports one cut off as possibly committed'. These run on real timers of a few hundred milliseconds, where upstream's fake timers do not wait. 'lets a locate outlive its budget' was already ported.
- Upstream's Compose Multiplatform text fields on iOS (`mobile reads Compose Multiplatform text fields on iOS`, `75ae2ab`, tester-army/e2e#1427) are ported in part. `SnapshotProjection.ProjectSnapshot` reads the placeholder an empty iOS field folds into its label (upstream's `foldedPlaceholder` in `nodes.ts`, `FoldedPlaceholder` here): for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out of the tree, and the name is the label before `, ` (a non-empty `ContentDescription` stays the name, and that label is then the `Text`). A field of any other shape keeps its label and children as reported. The private record `Folded` is a .NET-only name for upstream's inline `{ label, placeholder, child }` object. Test: 'reads the placeholder an empty Compose Multiplatform field folds into its label' in `Nodes/SnapshotProjectionTests`. Not ported: `mayTakeText` and `NodeBinding.takesText` (`nodes.ts`, `surface.ts`), with the three `it`s that use them in `describe('input freshness')` of `engine.test.ts` ('acts on an unclassified iOS field', 'rejects an unclassified iOS element', 'rejects an unclassified iOS element that gained a control before typing'). The port's `fill` and `clear` have no check of the node's role and send the text to any node with a ref, so an iOS `other` is already typed into; the check upstream widens does not exist here. Adding it would be a new refusal, not this change; it is tracked in #850.
- Upstream's WebDriver element-type fix (`fix(mobile): read XCUIElementType class names as their XCTest types`, `3f9a7d8`, tester-army/e2e#1133) is ported. `SnapshotProjection.NormalizeKind` is upstream's `normalizeKind` in `nodes.ts`: it drops `^XCUIElementType(?=[A-Z])` (`XcuiElementTypePrefix`, a .NET `[GeneratedRegex]` for upstream's inline regex) before it maps the type, so the `XCUIElementTypeButton` that agent-device's WebDriver runtime reports on hosted Appium clouds reads as `Button`. `DeviceSession.KeyboardShowing` is upstream's `keyboardShowing` in `surface.ts`: it routes node types through `NormalizeKind`, so an `XCUIElementTypeKeyboard` counts as a shown keyboard. Upstream records this only in its changeset; its skill references and docs say nothing about it, so `SKILL.md` and `docs/` have nothing to mirror. Tests: in `Nodes/SnapshotProjectionTests`, 'reads the XCUIElementType class names a WebDriver session reports as the native XCTest types'; in `Engine/DeviceFixtureDismissingAKeyboardThatHasNoDismissKeyTests`, 'reads an XCUIElementTypeKeyboard from a WebDriver session as a shown keyboard'.
- Upstream's held-device skip (`mobile: skip booted devices another agent-device session holds`, `9e53c40`, tester-army/e2e#1379) is ported. `BootedDevices` (`DevicePool.cs`, upstream's class in `pool.ts`) skips a booted device whose inventory entry has a `claimedBy.session` that is not one of this target's own slot sessions, logs the skipped ones, and fails `prepare` with a non-retryable `ENGINE_FAILURE` when every booted device is held. `BootedDevice` is upstream's interface of that name, as a record. `LifecycleTests.Skips_a_booted_device_another_agent_device_session_holds_keeps_one_its_own_slot_session_holds_and_fails_when_every_booted_one_is_held` ports the `lifecycle` `it` of `engine.test.ts`, whole.
- Not ported: the tool-pixels test (needs the full runner and a scripted model), and the `device.openApp` check of an unknown permission name.
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/get-started/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ public sealed class GreetingTests : MobileAppTest
}
```

`GetByTestId` finds a control by its accessibility identifier. `Device` is the device fixture: `HomeAsync`, `OpenAppAsync`, `ForegroundAppAsync`, and `SetAppearanceAsync(Appearance.Dark)`.
`GetByTestId` finds a control by its accessibility identifier. A field's hint is its `Placeholder`, shown or not, so `GetByPlaceholder` finds it. Compose Multiplatform reports no placeholder on iOS: an empty field appends the text it draws to its label (`Name, Ada Lovelace`) over one `StaticText` child. For an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text (`Name, Ada Lovelace`), that text is the field's `Placeholder`, the child is left out of the tree, and the name is the label before `, ` (a non-empty `ContentDescription` stays the name, and that label is then the `Text`). A field of any other shape keeps its label and children as reported. So `GetByPlaceholder("Ada Lovelace")` finds such a field and `GetByText("Ada Lovelace")` does not. `Device` is the device fixture: `HomeAsync`, `OpenAppAsync`, `ForegroundAppAsync`, and `SetAppearanceAsync(Appearance.Dark)`.

```csharp
await Device.HomeAsync();
Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ trip. The installed tool prints the same text with `e2e guide <topic>`
| Running tests: `dotnet test` filters, the `e2e` tool, `report.json` for scripts, exit codes, CI, test filters (not wired into `dotnet test`: the `ITestFilter` seam and the `run.changes` and `run.filters` report fields) | `running` | [references/running.md](references/running.md) |
| Handling a failed run: the trace page, `report.json` for scripts, `E2ESession.Report()` for a session you run yourself, error codes and their fixes, a visible browser, replay misses, phase timings, the agent step table, every model call | `debugging` | [references/debugging.md](references/debugging.md) |
| Exploring an app toward a goal without a test file, with `e2e explore` or `Explore.ExploreAsync` | `explore` | [references/explore.md](references/explore.md) |
| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app | `mobile` | [references/mobile.md](references/mobile.md) |
| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held), video of the device screen, `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted; unstable; `INVALID_STATE` after the session is disposed), Limrun hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children) | `mobile` | [references/mobile.md](references/mobile.md) |
| Testing a Windows desktop app: `WinAppEngine`, `WinAppTest`, the `Window` fixture, the `windows` target in `e2e explore` and `e2e mcp`, the interactive-desktop requirement, known limits | `windows` | [references/windows.md](references/windows.md) |
| Asked to bug bash or hunt for bugs: proving each bug with a repro test | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) |

Expand Down
1 change: 1 addition & 0 deletions skills/e2e-dotnet/references/mobile.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ Limits of the preview: the tests attach to the running app, so `Device.InstallAp

## Known limits

- Compose Multiplatform reports no placeholder on iOS: an empty field appends the text it draws to its label (`Name, Ada Lovelace`) over one `StaticText` child. For an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text (`Name, Ada Lovelace`), that text is the field's `Placeholder`, the child is left out of the tree, and the name is the label before `, ` (a non-empty `ContentDescription` stays the name, and that label is then the `Text`). A field of any other shape keeps its label and children as reported. So `GetByPlaceholder("Ada Lovelace")` finds such a field and `GetByText("Ada Lovelace")` does not.
- On iOS, a plain React Native `View` has no children in the tree, so `Filter(has: ...)` on it and queries scoped to it match nothing, while `Filter(hasNot: ...)` keeps every such row, including one you meant to exclude. Scope to the screen's `ScrollView` (`Device.Locator("role=scroll-area")`), or give the children test ids.
- `Locator.PressAsync` takes `Enter`, `Space`, or one character, sent through the soft keyboard. A field that is not focused is tapped first, for `Enter` as for a character, so the keyboard is up. Other named keys and every modifier fail with `UNSUPPORTED_CAPABILITY`.
- Android needs `adb` from the Android SDK Platform-Tools on `PATH`, locally and for the Limrun tunnel.
Expand Down
47 changes: 44 additions & 3 deletions src/E2E.Mobile/SnapshotProjection.cs
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,10 @@ SemanticNode Build(int position, ProjectedNode? parent)
}

var identifier = NonEmpty(source.Identifier);
var label = NonEmpty(source.Label);
IReadOnlyList<int> kids = children.TryGetValue(position, out var below) ? below : [];
var blank = editable && !secure && string.IsNullOrEmpty(value) && NonEmpty(source.Placeholder) is null;
var folded = !android && blank ? FoldedPlaceholder(source, kids, raw, children) : null;
var label = folded is null ? NonEmpty(source.Label) : folded.Label;
var name = NonEmpty(source.ContentDescription) ?? label;
var entry = new ProjectedNode
{
Expand All @@ -324,7 +327,7 @@ SemanticNode Build(int position, ProjectedNode? parent)
States = reported,
};
index.Add(entry);
var childNodes = children.TryGetValue(position, out var below) ? below.Select(child => Build(child, entry)).ToList() : [];
var childNodes = kids.Where(child => child != folded?.Child).Select(child => Build(child, entry)).ToList();

// A secure field's value is never observed; the tree carries that it is secure, not what it
// holds. Android echoes a text view's label as its value, which would render every line twice,
Expand Down Expand Up @@ -355,7 +358,7 @@ SemanticNode Build(int position, ProjectedNode? parent)
Checked = checkedState == true,
Expanded = expanded == true,
},
Placeholder = NonEmpty(source.Placeholder),
Placeholder = folded?.Placeholder ?? NonEmpty(source.Placeholder),

// The accessibility identifier (iOS) or resource id (Android) is the node's test id.
TestId = identifier,
Expand Down Expand Up @@ -601,6 +604,42 @@ private static bool ShowsHint(RawNode source, bool android, bool editable)
return !string.IsNullOrEmpty(placeholder) && source.Value?.Trim() == placeholder;
}

/// <summary>
/// The placeholder an empty iOS field folds into its label, with the child that draws it. Compose
/// Multiplatform reports no placeholder: it draws one as a text inside the field and appends that text to
/// the field's label, so an empty "Name" field with the example "Ada Lovelace" reads
/// <c>"Name, Ada Lovelace"</c> over one <c>StaticText</c> child. Null unless the field's only child is that
/// text and the field has a label of its own.
/// </summary>
private static Folded? FoldedPlaceholder(RawNode source, IReadOnlyList<int> kids, IReadOnlyList<RawNode> raw, Dictionary<int, List<int>> children)
{
if (kids.Count != 1)
{
return null;
}

var child = kids[0];
var text = raw[child];
if (children.ContainsKey(child))
{
return null;
}

var placeholder = NonEmpty(text.Label);
var label = source.Label;
if (KindOf(text) != "static-text" || placeholder is null || label is null)
{
return null;
}

if (!label.EndsWith($", {placeholder}", StringComparison.Ordinal) || label.Length == placeholder.Length + 2)
{
return null;
}

return new Folded(label[..^(placeholder.Length + 2)], placeholder, child);
}

/// <summary>The string itself, or null for an absent or empty one.</summary>
private static string? NonEmpty(string? value) => string.IsNullOrEmpty(value) ? null : value;

Expand Down Expand Up @@ -747,4 +786,6 @@ private sealed record SpelledStates(bool? Checked = null, bool? Expanded = null)

/// <summary>What React Native spelled into an iOS accessibility value.</summary>
private sealed record SpelledValue(string Role, SpelledStates States, string? Value);

private sealed record Folded(string Label, string Placeholder, int Child);
}
Loading
Loading