Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,11 +168,11 @@ Differences:
Differences:

- A session is one `Session` class with `Screen`, `App`, `Platform`, `Browser`, `SaveStateAsync`, `SetDefaultTimeout`, `CloseAsync`, and `DisposeAsync`, where upstream's is a flat object typed by the engine's fixtures (`Host<Fixtures>`). There is no `Device` member on it yet (#1678). It has no `agent`: `createAgent` is not ported (#1676).
- `LaunchOptions` has no `Headed` (the web engine takes its window mode through its own options), no `Signal` or `HandleSignals` (the token passed to `LaunchAsync` and `NewSessionAsync` is the signal; closing on SIGINT, SIGTERM, and SIGHUP is #1677), and `Timeout` is a `TimeSpan`. `Env` is a dictionary and `OnNotice` takes the line only.
- `LaunchOptions` has no `Headed` (the web engine takes its window mode through its own options), no `Signal` (the token passed to `LaunchAsync` and `NewSessionAsync` is the signal). `HandleSignals` (upstream's `handleSignals`, default true) closes the host on SIGINT, SIGTERM, and SIGHUP (`Core/Signals.cs`, upstream's `core/signals.ts` and its `closeOnSignals`): its handler closes every registered host, says a failed close on stderr as upstream does, and returns, so the runtime ends the process by the signal unless a `PosixSignalRegistration` of the program cancels it. Upstream raises the signal again itself and marks the listeners an engine registers while it opens (`engineListeners`); .NET has neither, since a handler that does not cancel ends the process by the signal and the runtime lists no registrations, so an engine's own listener cannot be told from the program's. Windows has no SIGHUP and its Ctrl-C/close map to the same registrations; the tests run on Unix, and `Timeout` is a `TimeSpan`. `Env` is a dictionary and `OnNotice` takes the line only.
- `RedactThrown` rebuilds an error that carries a secret value (type and code kept, causes redacted) where upstream edits the thrown error in place, which a .NET exception's read-only message does not allow. An error with no secret is thrown as it is.
- `SessionState.ExportAsync` and `ImportAsync` are `Task`-returning like upstream's `async` methods, so a bad key rejects instead of throwing. The key is `SessionStateKeyOptions.Key`, a `byte[]` of 32 bytes; there is no unknown-option check (`plaintext: true` cannot be passed).
- `app.linkBase` (`AppConfig.LinkBase`, `ResolvedApp.LinkBase`, `LaunchApp.LinkBase`, checked by `TargetApps.CheckLinkBase`, upstream's `checkLinkBase`) and the engine `openLink` hook (`IEngineSession.OpenLinkAsync`, `EngineSessionSpec.OpenLink`, `ISessionApp.OpenLinkAsync`, `session/openLink` in the manifest schema) are ported, with `App.OpenAsync(path)` joining and opening the link as upstream's `launchPinnedApp` does. `EngineCapabilities.OpenLink` is .NET-only: C# has no optional member to test, so an engine declares the hook the way it declares `Resize`. `TargetApps.JoinLink`, `TargetApps.RefusesLink`, and `Urls.IsRefusedLinkScheme` (upstream's `joinLink`, `refusesLink`, `isRefusedLinkScheme`) are shared with `DeviceLinks`. Not ported: the unknown-key message of a `defineEngine` session, which a typed `EngineSessionSpec` cannot receive (`engine.test.ts`).
- Tests: `Core/CoreSessionStateTests` is `core-state.test.ts`; `Core/CoreTests` is the ported part of `core.test.ts` (partial, and renamed `Partial_...`: 'opens sessions one at a time' (no `gadget` fixture or key list), 'refuses what is not an engine' (no non-engine or unknown app key to pass), 'says a closed session is closed' (the redaction half only), 'bounds each operation by its default timeout' (no agent step, which needs `createAgent`, #1676)); `AppLinkBase/` is `app-link-base.test.ts`; `Core/CoreTests` also holds 'opens a path as a link in the pinned app on a device target with a link base' (`Opens_a_path_as_a_link_...`); `SchemaFixtures/SessionStateV1SchemaTests`; `StepVerification/StepRecorderLastVerifiedStepIndexTests` has the two new `it` (`RedactThrown` is partial: the caught error is checked, not the original in place); `TraceCache/Trace1EntryTests` has the space-press `it`. Not ported: the `it` of `createAgent`, the link-base `it` of `core.test.ts`, `core-signals.test.ts`, and the `engine.test.ts` fixture-name check (#1676 to #1678).
- Tests: `Core/CoreSessionStateTests` is `core-state.test.ts`; `Core/CoreTests` is the ported part of `core.test.ts` (partial, and renamed `Partial_...`: 'opens sessions one at a time' (no `gadget` fixture or key list), 'refuses what is not an engine' (no non-engine or unknown app key to pass), 'says a closed session is closed' (the redaction half only), 'bounds each operation by its default timeout' (no agent step, which needs `createAgent`, #1676)); `AppLinkBase/` is `app-link-base.test.ts`; `Core/CoreTests` also holds 'opens a path as a link in the pinned app on a device target with a link base' (`Opens_a_path_as_a_link_...`); `SchemaFixtures/SessionStateV1SchemaTests`; `StepVerification/StepRecorderLastVerifiedStepIndexTests` has the two new `it` (`RedactThrown` is partial: the caught error is checked, not the original in place); `TraceCache/Trace1EntryTests` has the space-press `it`. `CoreSignals/CoreOnASignalTests` is `core-signals.test.ts`, run through `E2E.Tests.App core-signals` (the fake engine of the core tests in a child process); its SIGHUP `it` is `Partial_ends_by_SIGHUP_...`: the engine that listens for SIGHUP and the host's leaving that listener out are not ported (see `HandleSignals` above). Not ported: the `it` of `createAgent`, the link-base `it` of `core.test.ts`, and the `engine.test.ts` fixture-name check (#1676, #1678).

## Not ported

Expand Down
16 changes: 16 additions & 0 deletions docs-site/src/content/docs/api/E2E/LaunchOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,22 @@ The environment the engine reads; default the process environment.

**Value:** [IReadOnlyDictionary&lt;string, string&gt;](https://learn.microsoft.com/dotnet/api/system.collections.generic.ireadonlydictionary-2)

<a id="E2E_LaunchOptions_HandleSignals_"></a>
<a id="E2E_LaunchOptions_HandleSignals"></a>

### HandleSignals


```csharp
public bool HandleSignals { get; init; }
```


Closes the host on SIGINT, SIGTERM, and SIGHUP, so its browser or device session does not stay behind, then lets the signal end the program, unless
the program listens for it itself (a [PosixSignalRegistration](https://learn.microsoft.com/dotnet/api/system.runtime.interopservices.posixsignalregistration) that cancels the signal). Default true. Upstream's `handleSignals`.

**Value:** [bool](https://learn.microsoft.com/dotnet/api/system.boolean)

<a id="E2E_LaunchOptions_Headed_"></a>
<a id="E2E_LaunchOptions_Headed"></a>

Expand Down
3 changes: 2 additions & 1 deletion docs-site/src/content/docs/docs/core.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,13 @@ foreach (var sku in new[] { "A-1001", "B-2040" })
| Timeouts | An action, a wait, or a read gets 30 seconds. `App.OpenAsync()` and `App.RestartAsync()` get 120 seconds. `LaunchOptions.Timeout` or `session.SetDefaultTimeout(timeout)` sets one value for all of them, for the operations that start from then on. There is no session deadline. |
| Cancellation | The token passed to `LaunchAsync` and `NewSessionAsync` stops what is running. |
| App links | On a device, `LaunchApp.LinkBase` (an absolute URL or a scheme such as `shopapp://`, with no query or fragment) makes `session.App.OpenAsync("/orders/42")` open `shopapp://orders/42` in the pinned app. An absolute link opens as given; a `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, or `filesystem:` link is `POLICY_DENIED`. An engine with no link hook, such as `WebEngine`, refuses a link base with `INVALID_CONFIG`. |
| Ctrl-C | The host closes on SIGINT, SIGTERM, and SIGHUP, so its browser or device session does not stay behind. Then the program ends by the signal, unless it listens for the signal itself (a `PosixSignalRegistration` that sets `Cancel`). Turn it off with `LaunchOptions.HandleSignals = false`. |
| Window | `LaunchOptions.Headed` shows the browser window; the default is hidden. A device ignores it. |
| Files | Screenshots and downloads go to a temp directory that is deleted when the host closes. Set `ArtifactsDir` to keep them; a relative path resolves from `Cwd`. |
| Closing | `CloseAsync` is idempotent and throws an `AggregateException` of the failures closing found. |

<Aside type="note">
Not ported yet: closing the host on SIGINT, SIGTERM, and SIGHUP, and `createAgent`. Issues [#1676](https://github.com/hardkoded/e2e-dotnet/issues/1676) and [#1677](https://github.com/hardkoded/e2e-dotnet/issues/1677) track them.
Not ported yet: `createAgent`. Issue [#1676](https://github.com/hardkoded/e2e-dotnet/issues/1676) tracks it.
</Aside>

## Keep a signed-in state
Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ trip. The installed tool prints the same text with `e2e guide <topic>`
| Running tests: `dotnet test` filters, the `e2e` tool (`e2e doctor` checks that `e2e explore` and `e2e mcp` can start; `--json` prints the `doctor-1` report for a host, `--check-model` calls each model once; `e2e cache ls|stats|clear` lists, sizes, or empties the replay cache), `report.json` for scripts, exit codes, CI, test filters (NUnit, xUnit v3, and TUnit, mobile and Windows adapters included: `E2ERunFixture.Filters` and `ChangedSince`, or `E2E_CHANGED_SINCE`; `run.changes`, `run.filters`, and `filtered` skips in `report.json`), `--last-failed` reruns (`E2ERunFixture.LastFailed` or `E2E_LAST_FAILED=1`; `run.carried` in `report.json`, `FinishedRun.LastRun` for reporters, folded into the GitHub comment) | `running` | [references/running.md](references/running.md) |
| Handling a failed run: the trace page, `report.json` for scripts, `E2ESession.Report()` for a session you run yourself, error codes and their fixes, a visible browser, replay misses, phase timings, the agent step table, every model call | `debugging` | [references/debugging.md](references/debugging.md) |
| Exploring an app toward a goal without a test file, with `e2e explore` or `Explore.ExploreAsync` | `explore` | [references/explore.md](references/explore.md) |
| Driving an app from plain code with no test framework (`Launch.LaunchAsync(engine, new LaunchOptions { App, Secrets, Headed, Timeout, ArtifactsDir, Cwd })`, `Host.NewSessionAsync`, `Host.Secret`, `Session.SaveStateAsync`, and `SessionState.ExportAsync`/`ImportAsync` with a 32-byte key you keep), or running isolated per-test sessions from your own runner: `E2EHost.OpenAsync`, `OpenHostOptions` (config value or file, target, `EngineFactory`), `host.OpenSessionAsync`, `E2ESession.CloseAsync` and the failures it returns, `Steps()` | `host` | [references/host.md](references/host.md) |
| Driving an app from plain code with no test framework (`Launch.LaunchAsync(engine, new LaunchOptions { App, Secrets, Headed, Timeout, HandleSignals, ArtifactsDir, Cwd })`, `Host.NewSessionAsync`, `Host.Secret`, `Session.SaveStateAsync`, and `SessionState.ExportAsync`/`ImportAsync` with a 32-byte key you keep), or running isolated per-test sessions from your own runner: `E2EHost.OpenAsync`, `OpenHostOptions` (config value or file, target, `EngineFactory`), `host.OpenSessionAsync`, `E2ESession.CloseAsync` and the failures it returns, `Steps()` | `host` | [references/host.md](references/host.md) |
| Testing an iOS or Android app: `MobileEngine`, the `Device` fixture, picking a simulator (without `MobileOptions.Device`, `MobileEngine` skips booted devices another agent-device session holds, and `prepare` fails with `ENGINE_FAILURE` when every booted one is held, and with none booted agent-device boots a shut-down one: on iOS an iPhone over an iPad over an Apple TV, on Android a phone or tablet over a TV, then newest OS, then name order, and a reachable paired iPhone or iPad counts as booted; `prepare` logs a per-device timing line), video of the device screen, `Device.SetPermissionAsync` (the enums, or wire-name strings such as `"camera"` and `"grant"`; an unknown name or state is `INVALID_ARGUMENT` before any device command), `Device.UnstableRaw` (the test's agent-device session: name, device id and name, selection, and a raw `IDeviceClient` whose calls are not steps, replayed, or redacted, and that throws `NotSupportedException` for the `text-size` and `animations` settings, `apps.uninstall`, and `command.hardwareButton`; unstable; `INVALID_STATE` after the session is disposed; a custom agent tool gets the running attempt's from `MobileTools.UnstableDeviceSession(engines)`), a NativeScript `testID` or `accessibilityIdentifier` as the test id `GetByTestId` reads (an Android `accessibilityLabel` is put before the text), Limrun and BrowserStack hosted devices, a MAUI app through the DevFlow agent, known limits, `Locator.PressAsync("Enter")` tapping an unfocused field first, and the agent's `open_app` returning to the app under test when it names no app, a Compose Multiplatform field on iOS (for an empty (no value), editable, non-secure iOS field that reports no placeholder of its own, whose only child is a `StaticText` with no children, and whose label is a non-empty name, then `, `, then that child's text, that text is the field's `Placeholder`, the child is left out, and the name is the label before `, `; a non-empty `ContentDescription` stays the name, and that label is then the `Text`; any other field keeps its label and children), and `FillAsync` and `ClearAsync` refusing a node that is not an editable field with `UNSUPPORTED_CAPABILITY` (an iOS `other` that holds a value or only text is typed into), reading the field again before typing (a field the new capture no longer lists, an incomplete capture, or a changed input purpose or secure, disabled, or hidden state is a retryable `NODE_STALE` before anything is typed), looking at the screen again after the keyboard opens or closes (an unmoved control is acted on at once only when keyboard visibility is unchanged, else it waits out `TransitionMilliseconds`; a test's screen swipe waits out the remaining budget before it scrolls), and a `target_still_moving` refusal with `dispatched: no` being a retryable `NODE_STALE` | `mobile` | [references/mobile.md](references/mobile.md) |
| Testing a Windows desktop app: `WinAppEngine`, `WinAppTest`, the `Window` fixture, the `windows` target in `e2e explore` and `e2e mcp`, the interactive-desktop requirement, known limits | `windows` | [references/windows.md](references/windows.md) |
| Asked to bug bash or hunt for bugs: the bug-bash config, many `e2e explore` runs at once, proving each bug with a repro test | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) |
Expand Down
3 changes: 2 additions & 1 deletion skills/e2e-dotnet/references/host.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,4 +100,5 @@ await session.Screen.GetByLabel("Password").FillAsync(shop.Secret("SHOP_PASSWORD
- `LaunchOptions.Timeout` bounds an action, a wait, a read, and `App.OpenAsync()` or `App.RestartAsync()` (30 seconds for the first three and 120 seconds for the last two without it). `session.SetDefaultTimeout(timeout)` moves all of them for the operations that start from then on; a zero or infinite span is `INVALID_ARGUMENT`. There is no session deadline. `Headed` shows the browser window (a device ignores it). An operation on a closed session fails with `CANCELLED` and the message `the session is closed`. `ArtifactsDir` keeps screenshots and downloads (a relative path resolves from `Cwd`); without it they go to a temp directory deleted when the host closes.
- `await session.SaveStateAsync()` captures cookies and storage in memory. Pass it to `shop.NewSessionAsync(new NewSessionOptions { State = state })`. To keep it between runs, `state.ExportAsync(new SessionStateKeyOptions { Key = key })` with a 32-byte key you keep, and `SessionState.ImportAsync(bytes, options)`. A state restores only on the same engine, platform, and app (`SESSION_MISMATCH`; give two addresses of one app the same `LaunchApp.Identity`), a wrong key or changed byte is `SESSION_INVALID`, and an old one `SESSION_EXPIRED`.
- `LaunchApp.LinkBase` (an absolute URL or a scheme such as `shopapp://`, with no query or fragment) makes `session.App.OpenAsync("/orders/42")` open `shopapp://orders/42` in the pinned app of a device engine. An absolute link opens as given; `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, and `filesystem:` are `POLICY_DENIED`; an engine without a link hook, or a bad value, is `INVALID_CONFIG` at launch.
- Not ported yet: closing on signals and `createAgent` (issues 1676 to 1678).
- `LaunchOptions.HandleSignals` (default true) closes the host on SIGINT, SIGTERM, and SIGHUP, so its browser or device session does not stay behind, then lets the signal end the program. A program that listens for the signal itself (a `PosixSignalRegistration` that sets `Cancel`) decides instead. A close that fails is said on stderr. Set it to `false` to close the host yourself.
- Not ported yet: `createAgent` (issue 1676).
13 changes: 11 additions & 2 deletions src/E2E/Core/Launch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ public sealed record LaunchOptions
/// </summary>
public TimeSpan? Timeout { get; init; }

/// <summary>
/// Closes the host on SIGINT, SIGTERM, and SIGHUP, so its browser or device session does not stay behind, then lets the signal end the program, unless
/// the program listens for it itself (a <see cref="System.Runtime.InteropServices.PosixSignalRegistration"/> that cancels the signal). Default true. Upstream's <c>handleSignals</c>.
/// </summary>
public bool HandleSignals { get; init; } = true;

/// <summary>
/// Where screenshots, downloads, and recordings are written, from <see cref="Cwd"/>. Default: a fresh directory under the system temp directory,
/// deleted when the host closes.
Expand Down Expand Up @@ -195,9 +201,10 @@ public sealed class Host : IAsyncDisposable
private readonly Func<StandaloneAttempt, StateIdentity, Action, Session> _newSession;
private readonly Lock _gate = new();
private readonly Lazy<Task> _closing;
private readonly Action? _unregisterSignals;
private Session? _open;

internal Host(StandaloneHost host, IEngine engine, string target, string projectRoot, string output, bool ownsArtifacts, IReadOnlyDictionary<string, string> secrets, StateIdentity identity, TimeSpan? timeout, LaunchApp? app, bool headed, Func<StandaloneAttempt, StateIdentity, Action, Session>? newSession = null)
internal Host(StandaloneHost host, IEngine engine, string target, string projectRoot, string output, bool ownsArtifacts, IReadOnlyDictionary<string, string> secrets, StateIdentity identity, TimeSpan? timeout, LaunchApp? app, bool headed, bool handleSignals, Func<StandaloneAttempt, StateIdentity, Action, Session>? newSession = null)
{
_newSession = newSession ?? ((attempt, sessionIdentity, onClosed) => new Session(attempt, sessionIdentity, onClosed));
_headed = headed;
Expand All @@ -212,6 +219,7 @@ internal Host(StandaloneHost host, IEngine engine, string target, string project
_identity = identity;
_timeout = timeout;
_closing = new Lazy<Task>(CloseOnceAsync);
_unregisterSignals = handleSignals ? Signals.CloseOnSignals(CloseAsync) : null;
}

/// <summary>The engine's platform label: <c>web</c>, <c>ios</c>, <c>android</c>, or an engine's own string.</summary>
Expand Down Expand Up @@ -290,6 +298,7 @@ private void Release(StandaloneAttempt attempt)

private async Task CloseOnceAsync()
{
_unregisterSignals?.Invoke();
var errors = new List<Exception>();
Session? open;
lock (_gate)
Expand Down Expand Up @@ -484,7 +493,7 @@ private static async Task<Host> LaunchCoreAsync(IEngine engine, LaunchOptions? o
Headed = options.Headed,
Notice = options.OnNotice is { } notice ? (_, message) => OpenSession.Notify(notice, message) : null,
}).ConfigureAwait(false);
return new Host(worker, engine, target, projectRoot, output, ownsArtifacts, secrets, identity, options.Timeout, app, options.Headed, newSession);
return new Host(worker, engine, target, projectRoot, output, ownsArtifacts, secrets, identity, options.Timeout, app, options.Headed, options.HandleSignals, newSession);
}
catch
{
Expand Down
Loading
Loading