From 551cc55c892de6c01d2a90a4da3c7c667075d093 Mon Sep 17 00:00:00 2001 From: DongYun Kang Date: Fri, 13 Nov 2026 15:45:57 +0900 Subject: [PATCH 1/7] feat(core): add launch, SessionState, and createAgent to @e2e-dev/core @e2e-dev/core becomes usable on its own. launch(engine) opens a core attempt on the standalone worker (nothing published to globalThis, no process secret registry, no trace); sessions save and restore session-state-1 state, AES-256-GCM encrypted under a caller's key, and take secrets from launch(). createAgent from @e2e-dev/core/agent adds the agent with a replay cache keyed by cacheKey and a guard the dispatcher checks before every action, replayed ones included. @e2e-dev/web/engine and @e2e-dev/mobile/engine export the factories without test, so the engines' e2e peer becomes optional. Device targets take app.linkBase, opened through session.openLink. Co-Authored-By: Claude Opus 5.5 --- .changeset/e2e-core.md | 14 + AGENTS.md | 30 +- CONTRIBUTING.md | 11 +- docs/core.mdx | 252 ++++++++++++ docs/docs.json | 3 +- docs/examples/core/checkout.ts | 35 ++ docs/examples/core/prices.ts | 12 + docs/mobile.mdx | 2 +- docs/package.json | 1 + docs/reference/agent.mdx | 23 +- docs/reference/app.mdx | 14 +- docs/reference/config.mdx | 25 ++ docs/reference/engine-protocol.mdx | 1 + docs/reference/engine.mdx | 41 +- docs/reference/errors.mdx | 16 +- docs/reference/mobile.mdx | 7 +- docs/reference/security.mdx | 2 + docs/writing-an-engine.mdx | 4 +- packages/core/README.md | 31 +- packages/core/package.json | 10 +- packages/core/src/agent/act.ts | 34 +- packages/core/src/agent/action-dispatcher.ts | 150 ++++++- packages/core/src/agent/error.ts | 2 +- packages/core/src/agent/guard.ts | 168 ++++++++ packages/core/src/agent/invocation.ts | 5 +- packages/core/src/agent/step-accounting.ts | 4 +- packages/core/src/agent/step-cache.ts | 20 +- packages/core/src/cache/recorder.ts | 28 ++ packages/core/src/cache/trace-read.ts | 46 ++- packages/core/src/cache/trace.ts | 27 +- packages/core/src/config/app.ts | 58 ++- packages/core/src/core/agent.ts | 179 +++++++++ packages/core/src/core/index.ts | 33 ++ packages/core/src/core/internals.ts | 65 ++++ packages/core/src/core/launch.ts | 288 ++++++++++++++ packages/core/src/core/signals.ts | 49 +++ packages/core/src/core/state.ts | 221 +++++++++++ packages/core/src/engine/index.ts | 21 +- packages/core/src/engine/manifest.ts | 8 +- packages/core/src/engine/protocol/protocol.ts | 1 + packages/core/src/engine/session.ts | 1 + packages/core/src/engine/surface.ts | 2 + packages/core/src/internal/brands.ts | 2 + packages/core/src/internal/urls.ts | 12 + packages/core/src/locator/engine.ts | 15 +- packages/core/src/run/engine-sessions.ts | 17 +- packages/core/src/run/fixture-recording.ts | 8 +- packages/core/src/run/fixtures.ts | 28 +- packages/core/src/run/redact-opening.ts | 24 ++ packages/core/src/run/standalone-attempt.ts | 14 +- packages/core/src/run/standalone.ts | 30 +- packages/core/src/types.ts | 96 ++++- packages/core/tests/unit/entries.test.ts | 14 +- .../e2e/schema/engine-manifest-v1.schema.json | 1 + .../fixtures/session-state-v1.invalid.json | 18 + .../fixtures/session-state-v1.valid.json | 19 + .../e2e/schema/session-state-v1.schema.json | 58 +++ packages/e2e/src/agent/public.ts | 3 +- packages/e2e/src/config/resolve-run.ts | 5 +- packages/e2e/src/run/open-session.ts | 24 +- packages/e2e/src/run/standalone-kinds.ts | 2 + packages/e2e/tests/helpers/fake-engine.ts | 16 +- .../tests/integration/core-signals.test.ts | 60 +++ packages/e2e/tests/integration/core.test.ts | 368 ++++++++++++++++++ packages/e2e/tests/types/sdk-types.ts | 42 ++ packages/e2e/tests/unit/app-link-base.test.ts | 32 ++ packages/e2e/tests/unit/core-guard.test.ts | 115 ++++++ packages/e2e/tests/unit/core-state.test.ts | 66 ++++ packages/e2e/tests/unit/engine.test.ts | 2 +- .../e2e/tests/unit/package-imports.test.ts | 9 +- packages/e2e/tests/unit/scheduler.test.ts | 2 + packages/e2e/tests/unit/serial-record.test.ts | 2 + packages/e2e/tests/unit/trace-cache.test.ts | 9 + .../e2e/tests/unit/trace-template.test.ts | 10 + packages/e2e/tests/unit/units.test.ts | 2 + packages/mobile/README.md | 2 + packages/mobile/package.json | 7 + packages/mobile/src/engine-entry.ts | 16 + packages/mobile/src/engine.ts | 12 +- packages/mobile/src/fixtures.ts | 8 + packages/mobile/src/index.ts | 16 +- packages/mobile/src/links.ts | 6 +- packages/mobile/src/surface.ts | 11 +- packages/mobile/src/tools.ts | 2 +- packages/mobile/tests/unit/engine.test.ts | 17 +- packages/web/README.md | 8 + packages/web/package.json | 9 + packages/web/src/browser.ts | 2 +- packages/web/src/engine-entry.ts | 40 ++ packages/web/src/engine.ts | 5 +- packages/web/src/evaluation.ts | 2 +- packages/web/src/fixtures.ts | 16 + packages/web/src/index.ts | 46 +-- .../skills/e2e/references/writing-tests.md | 10 +- pnpm-lock.yaml | 3 + scripts/check-docs-examples.ts | 2 + skills/e2e/references/writing-tests.md | 10 +- 97 files changed, 3087 insertions(+), 232 deletions(-) create mode 100644 .changeset/e2e-core.md create mode 100644 docs/core.mdx create mode 100644 docs/examples/core/checkout.ts create mode 100644 docs/examples/core/prices.ts create mode 100644 packages/core/src/agent/guard.ts create mode 100644 packages/core/src/core/agent.ts create mode 100644 packages/core/src/core/index.ts create mode 100644 packages/core/src/core/internals.ts create mode 100644 packages/core/src/core/launch.ts create mode 100644 packages/core/src/core/signals.ts create mode 100644 packages/core/src/core/state.ts create mode 100644 packages/core/src/run/redact-opening.ts create mode 100644 packages/e2e/schema/fixtures/session-state-v1.invalid.json create mode 100644 packages/e2e/schema/fixtures/session-state-v1.valid.json create mode 100644 packages/e2e/schema/session-state-v1.schema.json create mode 100644 packages/e2e/tests/integration/core-signals.test.ts create mode 100644 packages/e2e/tests/integration/core.test.ts create mode 100644 packages/e2e/tests/unit/app-link-base.test.ts create mode 100644 packages/e2e/tests/unit/core-guard.test.ts create mode 100644 packages/e2e/tests/unit/core-state.test.ts create mode 100644 packages/mobile/src/engine-entry.ts create mode 100644 packages/mobile/src/fixtures.ts create mode 100644 packages/web/src/engine-entry.ts create mode 100644 packages/web/src/fixtures.ts diff --git a/.changeset/e2e-core.md b/.changeset/e2e-core.md new file mode 100644 index 0000000000..96b014cf96 --- /dev/null +++ b/.changeset/e2e-core.md @@ -0,0 +1,14 @@ +--- +'e2e': patch +'@e2e-dev/core': patch +'@e2e-dev/web': patch +'@e2e-dev/mobile': patch +--- + +New package `@e2e-dev/core`: drive a browser or a native app from plain code, with no test runner, config file, or report. `launch(web())` prepares an engine and returns a host; `host.newSession()` opens an isolated session holding `screen`, `app`, `platform`, and the engine's fixture. `session.saveState()` captures state in memory, and `SessionState.export({ key })` returns it only AES-256-GCM encrypted, as `session-state-1` bytes. Secrets go to `launch({ secrets })` and fill through `host.secret(name)`. `createAgent(session, { model, cache, guard })` from `@e2e-dev/core/agent` adds the agent: with `cache`, every `act` names its recording with `cacheKey`; `guard` keeps actions for your code, and `act` then ends with `status: 'stopped'` before an action a `stop` rule matches, replayed actions included. + +`@e2e-dev/web` and `@e2e-dev/mobile` now need `e2e` only for their main entry, which adds the engine's fixture to `test`, so their `e2e` peer is optional. `@e2e-dev/core/agent` also exports what `e2e/agent` does, `defineTool` among it. + +Device targets take `app.linkBase`: `app.open('/orders/42')` then opens the path as a link in the pinned app, through the new optional engine hook `session.openLink` (protocol method `session/openLink`). `@e2e-dev/mobile` implements it. + +`@e2e-dev/web/engine` and `@e2e-dev/mobile/engine` export the engine factories and their types without `test`, so a program that only drives an engine never loads the test runner. `defineEngine()` and `EngineHandle` now carry the fixtures an engine contributes as a type: `web()` is an `EngineHandle`, `mobile()` an `EngineHandle`. `ActResult` gains `status` and `stoppedAt`, as a union a check on `status` narrows. `@e2e-dev/core/engine` (and `e2e/engine`) exports `isRefusedLinkScheme`, the scheme list app links are refused by, which `@e2e-dev/mobile` now uses for `device.openLink`. diff --git a/AGENTS.md b/AGENTS.md index e1d50d8519..2586c69034 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,7 +9,8 @@ There is no separate spec. The code is the contract, pinned in three places: - The emitted `packages/e2e/dist/index.d.ts` (and `dist/engine/index.d.ts`, `dist/runner.d.ts`, `dist/oauth/*.d.ts`), and `@e2e-dev/core`'s - `packages/core/dist/engine/index.d.ts`, are the public API. `packages/e2e/tests/types/sdk-types.ts` holds compile-time + `packages/core/dist/core/*.d.ts` and `dist/engine/index.d.ts`, + are the public API. `packages/e2e/tests/types/sdk-types.ts` holds compile-time assertions (`@ts-expect-error` lines) for the parts that are easy to loosen by accident; it runs under the package `typecheck`, never under vitest. - Wire formats live in `packages/e2e/schema/*.schema.json` (report-1, @@ -42,10 +43,16 @@ suites that consume the built packages the way a user would. the replay cache (`src/cache/`), secrets, and the session half of a worker: `run/engine-sessions.ts` (engine init and dispose, each attempt's session from launch to close, which the runner's `TargetExecutor` - extends), `run/standalone.ts` (the worker a runner host or MCP keeps - alive, with a `WorkerPolicy` per kind), and `run/standalone-attempt.ts` - (one attempt: reservation, launch, fixtures, deadline, close). Its one - public entry is `@e2e-dev/core/engine`. It imports nothing of `e2e`: + extends), `run/standalone.ts` (the worker an `@e2e-dev/core` host, a runner + host, or MCP keeps alive, with a `WorkerPolicy` per kind), and + `run/standalone-attempt.ts` (one attempt: reservation, launch, fixtures, + deadline, close). `src/core/` is its own surface: `launch` opens a + `core` attempt (nothing published to `globalThis`, no process secret + registry, no trace, no session deadline), `SessionState` the + `session-state-1` export, and `createAgent` (`@e2e-dev/core/agent`) the agent + with a per-`cacheKey` replay cache settled at session close and a `guard` + (`src/agent/guard.ts`) the action dispatcher checks before every action, + replayed ones included. It imports nothing of `e2e`: `tests/unit/package-imports.test.ts` in `packages/e2e` fails on a dependency `@e2e-dev/core`'s manifest does not declare. `e2e` reaches its modules as `@e2e-dev/core/lib/` (`./lib/*` in its exports), which is @@ -93,8 +100,11 @@ suites that consume the built packages the way a user would. checks that need a stored login and are never part of `pnpm test`. - `packages/web` — the published `@e2e-dev/web` package: the browser engine, built with the public `defineEngine`, contributing the - `browser` fixture and `expect(browser)`. It depends on `@e2e-dev/core` and - `e2e` (peers), never the reverse; a target names it explicitly as + `browser` fixture and `expect(browser)`. `@e2e-dev/web/engine` (and + `@e2e-dev/mobile/engine`) export the factory and types without `test`, so + `@e2e-dev/core` programs never import `e2e`'s main entry. It depends on + `@e2e-dev/core` (peer), never the reverse, and on `e2e` as an optional peer + that only the main entry loads; a target names it explicitly as `engine: web()`. There is no default engine and no well-known id registry in core. It imports from `@e2e-dev/core/engine` only: the semantics every engine must reproduce (error taxonomy, text and URL matching, assertion @@ -103,7 +113,7 @@ suites that consume the built packages the way a user would. - `packages/mobile` — the published `@e2e-dev/mobile` package: the iOS/Android engine on agent-device, built with the same public `defineEngine`, contributing the `device` fixture. Like the web engine it - depends on `@e2e-dev/core` and `e2e` (peers), never the reverse; the + depends on `@e2e-dev/core` (peer) and optionally `e2e`, never the reverse; the engine implementation imports from `@e2e-dev/core/engine` only, and a target names it as `engine: mobile()`. The `@e2e-dev/mobile/tools` subpath holds the agent-side `open_app`, `swipe`, @@ -367,7 +377,7 @@ pnpm --filter @e2e-dev/testbed run test:headed - Package `test` scripts do **not** build. Root `build` and `test` order the packages explicitly rather than relying on topological sort, because `e2e` devDepends on the web engine for its browser-backed integration tests - while the engine peer-depends on `e2e` — pnpm reports that cycle + while the engine optionally peer-depends on `e2e` — pnpm reports that cycle on every install. `@e2e-dev/core` builds first: `e2e` and every engine compile against its `dist`. - `pnpm typecheck` runs `build` first, then per-package `typecheck`. The @@ -713,7 +723,7 @@ trees, on both platforms, without a device. entry. On `0.x` the bump is `patch` for a feature or fix; never pick `minor` on your own. A breaking change is a `minor` with the summary opening `**Breaking:**`; a `minor` or `major` for a release the maintainer - calls opens `**Release:**` (`scripts/check-changeset-bumps.ts`, `pnpm check`). Peer ranges point one way only (engine -> `@e2e-dev/core` and `e2e`, integration -> + calls opens `**Release:**` (`scripts/check-changeset-bumps.ts`, `pnpm check`). Peer ranges point one way only (engine -> `@e2e-dev/core` and optionally `e2e`, integration -> engine) and read `>= `, the floor the oldest sibling the package works with. Raise it when a package starts relying on a newer sibling's export or contract field; diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 045e703bd8..1676e087fe 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -289,9 +289,10 @@ committed. The following are the public contract. A change that breaks any of them is a breaking change and follows the rules below: -- Public types exported from `e2e`, `e2e/engine`, and `@e2e-dev/core/engine`, - as emitted in `dist/index.d.ts` and `dist/engine/index.d.ts` of `e2e`, - and `dist/engine/index.d.ts` of `@e2e-dev/core`. Removing an +- Public types exported from `e2e`, `e2e/engine`, `@e2e-dev/core`, + `@e2e-dev/core/agent`, and `@e2e-dev/core/engine`, as emitted in `dist/index.d.ts` + and `dist/engine/index.d.ts` of `e2e`, and `dist/core/*.d.ts` and + `dist/engine/index.d.ts` of `@e2e-dev/core`. Removing an export, narrowing an accepted input, widening a returned type, or changing a fixture's runtime behavior all count. - Config keys in `e2e.config.ts`, their types, and their defaults. @@ -335,8 +336,8 @@ together in one release, with a changeset for each, so an engine and a runner from the same release always match. `e2e` and `@e2e-dev/core` are a changesets `fixed` group: they always share a version, and `e2e` depends on that exact `@e2e-dev/core`. Engines declare a peer range on `@e2e-dev/core` that points one way -only (engine to core, `>=x <1`), plus one on `e2e` for the main entry that -adds the engine's fixture to `test`, and each integration (`@e2e-dev/kernel`, `@e2e-dev/lightpanda`, `@e2e-dev/eas`, `@e2e-dev/smol`, `@e2e-dev/testmu`, `@e2e-dev/browserstack`, `@e2e-dev/e2b`) does +only (engine to core, `>=x <1`), plus an optional one on `e2e` for the main +entry that adds the engine's fixture to `test`, and each integration (`@e2e-dev/kernel`, `@e2e-dev/lightpanda`, `@e2e-dev/eas`, `@e2e-dev/smol`, `@e2e-dev/testmu`, `@e2e-dev/browserstack`, `@e2e-dev/e2b`) does the same on the engines it plugs into; do not make it mutual or narrow it. ### Driver nightly compatibility diff --git a/docs/core.mdx b/docs/core.mdx new file mode 100644 index 0000000000..175011b48f --- /dev/null +++ b/docs/core.mdx @@ -0,0 +1,252 @@ +--- +title: "@e2e-dev/core" +description: Drive a browser or a native app from plain code, with an optional agent and no test runner. +--- + +`@e2e-dev/core` drives a browser or an iOS or Android app from plain code. It +uses the e2e engines, `screen`, and the agent, without the test runner: there +is no config file, no report, and no `expect`. Use it for automation, +scrapers, and bots. + +## Install + + + +```bash npm +npm install @e2e-dev/core @e2e-dev/web +``` + +```bash pnpm +pnpm add @e2e-dev/core @e2e-dev/web +``` + +```bash bun +bun add @e2e-dev/core @e2e-dev/web +``` + + + +Install `@e2e-dev/mobile` instead of, or beside, `@e2e-dev/web` to drive a +device. `@e2e-dev/core` needs no `e2e`, the test runner, which is built on it. +Import an engine from its `/engine` entry: the main entry adds the engine's +fixture to `test` and loads `e2e`. + +## Drive a browser + +`launch` prepares an engine and returns a host. A session is one isolated +browser context on that host, and it holds `screen`, `app`, `platform`, and +the engine's own fixture, here `browser`. + +```ts title="prices.ts" +import { launch } from '@e2e-dev/core'; +import { web } from '@e2e-dev/web/engine'; + +await using shop = await launch(web(), { app: { url: 'https://shop.example.com' } }); + +for (const sku of ['A-1001', 'B-2040']) { + await using session = await shop.newSession(); + await session.app.open(`/products/${sku}`); + if (await session.screen.getByText('Sold out', { exact: false, visible: true }).isVisible()) continue; + const price = await session.screen.getByTestId('sale-price').textContent(); + console.log(sku, price); +} +``` + +`screen` and `app` are the same API as in a test: see +[screen](/reference/screen) and [app](/reference/app). `await using` closes +each session at the end of its block, and the host, with its browser, at the +end of the file. + +| Rule | Behavior | +| --- | --- | +| Ownership | Each `launch` takes its own engine instance: `launch(web())`. A second host on the same instance fails with `WORKER_PROTOCOL`. | +| Sessions | One session at a time on a host. Close it before `newSession()`. For parallel work, launch more hosts. | +| Timeouts | Each operation has a default timeout: 30000 ms for actions, 5000 ms for waits. Set it with `launch({ timeout })` or `session.setDefaultTimeout(ms)`. There is no session deadline. | +| Cancellation | `launch({ signal })` and `newSession({ signal })` stop what is running. | +| Ctrl-C | The host closes on SIGINT, SIGTERM, and SIGHUP, so its browser or device session does not stay behind. Turn it off with `handleSignals: false`. | +| Files | Screenshots and downloads go to a temp directory that is deleted when the host closes. Set `artifactsDir` to keep them. | + +## Drive an app on a device + +Pass `mobile()` and the app. The same `screen` code runs on a browser and +on a device. + +```ts +import { launch } from '@e2e-dev/core'; +import { mobile } from '@e2e-dev/mobile/engine'; + +await using shopApp = await launch(mobile({ platform: 'ios' }), { + app: { bundleId: 'com.example.shop', linkBase: 'https://shop.example.com' }, +}); +await using session = await shopApp.newSession(); +await session.app.open('/products/A-1001'); +``` + +`app.linkBase` makes `app.open(path)` open a link in the app: +`https://shop.example.com/products/A-1001` as a universal link here, or +`shopapp://products/A-1001` with `linkBase: 'shopapp://'`. Without it, +`app.open()` launches the app and takes no path. `linkBase` also works in an +`e2e.config.ts` target; see [config](/reference/config#linkbase). + +## Keep a signed-in state + +`session.saveState()` captures a browser session's cookies and storage in +memory. Start a later session from it with `newSession({ state })`. The mobile +engine has no state to capture, so there it fails with +`UNSUPPORTED_CAPABILITY`. + +```ts +const state = await session.saveState(); +await using next = await shop.newSession({ state }); +``` + +To keep a state between program runs, encrypt it with a 32-byte key that you +keep, for example in a secret store. There is no plaintext export. + +```ts +import { randomBytes } from 'node:crypto'; +import { SessionState } from '@e2e-dev/core'; + +const key = randomBytes(32); +const bytes = await state.export({ key }); // write these to disk +const restored = await SessionState.import(bytes, { key }); +``` + +A state restores only on a host of the same engine, platform, and app. Another +is `SESSION_MISMATCH`. A wrong key or a changed byte is `SESSION_INVALID`. +The encrypted state also holds the secret values the session knew, so a host +launched without them still redacts them from what `screen` reads. + +## Pass secrets + +Give secret values to `launch`, then fill them by name. The values never +appear in errors or in what `screen` reads. + +```ts +await using shop = await launch(web(), { + app: { url: 'https://shop.example.com' }, + secrets: { SHOP_PASSWORD: process.env.SHOP_PASSWORD! }, +}); +await session.screen.getByLabel('Password').fill(shop.secret('SHOP_PASSWORD')); +``` + +An agent gets a secret through its params, by name only: +`agent.act('Sign in with {password}', { params: { password: shop.secret('SHOP_PASSWORD') } })`. + +## Let the agent act + +`createAgent` from `@e2e-dev/core/agent` puts an agent over one session. Its +`act`, `assert`, `waitFor`, and `extract` work as they do in a +[test](/reference/agent). + +```ts title="checkout.ts" +import { launch, isE2EError } from '@e2e-dev/core'; +import { createAgent } from '@e2e-dev/core/agent'; +import { web } from '@e2e-dev/web/engine'; +import { gateway } from 'ai'; +import { z } from 'zod'; + +await using shop = await launch(web(), { app: { url: 'https://shop.example.com' } }); +await using session = await shop.newSession(); +await session.app.open('/products/A-1001'); + +const agent = createAgent(session, { + model: gateway('anthropic/claude-sonnet-5-5'), + cache: { dir: '.shop-bot/cache' }, + guard: { stop: [{ role: 'button', name: /pay|place order/i }] }, +}); + +const step = await agent.act('Add the item to the cart and apply the coupon with the largest discount', { + cacheKey: 'cart-best-coupon', +}); +if (step.status !== 'stopped') throw new Error(`did not reach the payment button: ${step.summary}`); +// A check after the step lets the cache keep its recording. +await session.screen.getByRole('button', /pay now/i).waitFor(); + +const order = await agent.extract('the order total', { schema: z.object({ total: z.number() }) }); +if (order.total <= 39_000) { + try { + await session.screen.getByRole('button', /pay now/i).tap(); + } catch (error) { + // The click may have reached the server: check the order list, never retry the payment. + if (isE2EError(error) && error.code === 'ACTION_MAY_HAVE_COMMITTED') console.error('check /my/orders'); + throw error; + } +} else { + console.log(`the total, ${order.total}, is over the limit: the order stays unpaid`); +} +``` + +`createAgent` takes the options of an [`agents` entry](/reference/config#agents), +such as `model`, `judge`, `maxSteps`, and `context`, plus `cache` and `guard`. +There is no default model: an agent without one fails its first call with +`MODEL_UNAVAILABLE`. + +### Replay recorded steps + +With `cache`, `act` records each flow and replays it on the next run with no +model call. Every `act` then names its recording with `cacheKey`, or runs live +with `cache: false`. The type system and the runtime both refuse an `act` +without one. + +A recording is keyed by the `cacheKey`, the instruction, its params, and how +many times the session made the same call. A recording is saved when the +session closes, or its host does, and only when a check passed after the +step, such as `locator.waitFor()`, `agent.assert`, or `agent.waitFor`. When +an agent step fails, recordings not yet confirmed are dropped. A failure your +code catches, such as a `waitFor` timeout, drops nothing. Read +[Replay cache](/cache) for how a replay checks the screen and hands off to the +model. + +| Option | Default | Meaning | +| --- | --- | --- | +| `dir` | `.e2e/cache` | The directory, from the current directory. | +| `mode` | `read-write` | `read-only` replays and never writes. | +| `strict` | `false` | A recording that no longer replays fails with `REPLAY_STALE`. | +| `store` | - | A custom [`CacheStore`](/cache) in place of the files. | + +### Keep actions for your code + +`guard` names the actions the agent may not take on its own. In `act`, the +guard checks every action, replayed ones too, before the action runs. + +| Field | Effect | +| --- | --- | +| `stop` | The step ends before the action, with `status: 'stopped'` and the action in `stoppedAt`. Your code takes over. | +| `deny` | The action fails with `POLICY_DENIED`. The model reads the refusal and plans around it. | +| `check` | A function that gets each action the rules allowed and returns `'allow'`, `'deny'`, or `'stop'`. | +| `unidentified` | An action whose target the guard cannot name: `'deny'` (the default when a rule or `check` is set) or `'allow'`. | + +A rule matches the node the action lands on and the screen's location. Set +any of `role`, `name`, `text`, `testId`, `url`, `destination` (where a +`navigate` goes), `key`, and `actions`; every field you set must match. +`name` matches exactly unless it is a RegExp. + +An unidentified action is a point action (`tapAt`, `hoverAt`, `dragAt`) that +lands on no control the screen lists, such as a canvas, or a mutating +[project tool](/tools), which `actions` names as `tool:`. The guard +does not check `agent.assert`, which only reads and scrolls. A drag is +checked on the node it starts from, not where it drops. A recording made +under one set of rules never replays under another. + + +A rule on a button does not see Enter pressed in a field, which can submit +the form. Add a rule such as `{ key: 'Enter', url: /checkout/ }`. + + +## Next + + + + Every query and action a session holds. + + + What `act`, `assert`, `waitFor`, and `extract` take and return. + + + Choose and configure the model the agent calls. + + + How an engine works, and how to write one. + + diff --git a/docs/docs.json b/docs/docs.json index c9e8ea0a7d..34ad17da01 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -74,7 +74,8 @@ "pages": [ "web", "mobile", - "api-testing" + "api-testing", + "core" ] }, { diff --git a/docs/examples/core/checkout.ts b/docs/examples/core/checkout.ts new file mode 100644 index 0000000000..10b7b2b245 --- /dev/null +++ b/docs/examples/core/checkout.ts @@ -0,0 +1,35 @@ +import { launch, isE2EError } from '@e2e-dev/core'; +import { createAgent } from '@e2e-dev/core/agent'; +import { web } from '@e2e-dev/web/engine'; +import { gateway } from 'ai'; +import { z } from 'zod'; + +await using shop = await launch(web(), { app: { url: 'https://shop.example.com' } }); +await using session = await shop.newSession(); +await session.app.open('/products/A-1001'); + +const agent = createAgent(session, { + model: gateway('anthropic/claude-sonnet-5-5'), + cache: { dir: '.shop-bot/cache' }, + guard: { stop: [{ role: 'button', name: /pay|place order/i }] }, +}); + +const step = await agent.act('Add the item to the cart and apply the coupon with the largest discount', { + cacheKey: 'cart-best-coupon', +}); +if (step.status !== 'stopped') throw new Error(`did not reach the payment button: ${step.summary}`); +// A check after the step lets the cache keep its recording. +await session.screen.getByRole('button', /pay now/i).waitFor(); + +const order = await agent.extract('the order total', { schema: z.object({ total: z.number() }) }); +if (order.total <= 39_000) { + try { + await session.screen.getByRole('button', /pay now/i).tap(); + } catch (error) { + // The click may have reached the server: check the order list, never retry the payment. + if (isE2EError(error) && error.code === 'ACTION_MAY_HAVE_COMMITTED') console.error('check /my/orders'); + throw error; + } +} else { + console.log(`the total, ${order.total}, is over the limit: the order stays unpaid`); +} diff --git a/docs/examples/core/prices.ts b/docs/examples/core/prices.ts new file mode 100644 index 0000000000..21bff06431 --- /dev/null +++ b/docs/examples/core/prices.ts @@ -0,0 +1,12 @@ +import { launch } from '@e2e-dev/core'; +import { web } from '@e2e-dev/web/engine'; + +await using shop = await launch(web(), { app: { url: 'https://shop.example.com' } }); + +for (const sku of ['A-1001', 'B-2040']) { + await using session = await shop.newSession(); + await session.app.open(`/products/${sku}`); + if (await session.screen.getByText('Sold out', { exact: false, visible: true }).isVisible()) continue; + const price = await session.screen.getByTestId('sale-price').textContent(); + console.log(sku, price); +} diff --git a/docs/mobile.mdx b/docs/mobile.mdx index a087f59dd6..b29ff2aa00 100644 --- a/docs/mobile.mdx +++ b/docs/mobile.mdx @@ -779,7 +779,7 @@ export function cloudDevices(options: { provider: string; device: string; osVers | On the web | On a device | | --- | --- | -| `app.open('/path')` and the agent's `navigate` verb | No app URL and no `navigate`. `app.open()` launches `app.bundleId` fresh, and `device.openLink(url)` opens a link into it | +| `app.open('/path')` and the agent's `navigate` verb | No app URL and no `navigate`. `app.open()` launches `app.bundleId` fresh, and `device.openLink(url)` opens a link into it. With [`app.linkBase`](/reference/config#linkbase), `app.open('/path')` opens the path as a link | | `test.setup` saves a session, `session:` restores it | Not available. A simulator has no portable session snapshot; sign in per test or seed the app | | Secret input | `type_secret` and `screen.getByLabel(...).fill(secret)` fill declared secrets into secure fields | | `--video` writes a WebM screencast | `--video` writes an MP4 of the device screen with taps shown (`videoTouches: false` hides them), or keeps a device provider's own recording | diff --git a/docs/package.json b/docs/package.json index 8cfe2d7e9c..ceef7af1f7 100644 --- a/docs/package.json +++ b/docs/package.json @@ -11,6 +11,7 @@ "@ai-sdk/typesafe-ai": "3.0.40", "@e2e-dev/browser-run": "workspace:*", "@e2e-dev/browserstack": "workspace:*", + "@e2e-dev/core": "workspace:*", "@e2e-dev/decision": "workspace:*", "@e2e-dev/e2b": "workspace:*", "@e2e-dev/limrun": "workspace:*", diff --git a/docs/reference/agent.mdx b/docs/reference/agent.mdx index c5409048a9..c4ca057b19 100644 --- a/docs/reference/agent.mdx +++ b/docs/reference/agent.mdx @@ -56,9 +56,11 @@ const { summary, cache } = await agent.act('Sign in', { ``` - `instruction` is 1 through 8192 UTF-8 bytes after NFC normalization. -- The [`ActResult`](#types) carries the step's `summary`, the `modelCalls` - and `actions` it spent, and `cache`: how the [replay cache](/cache) took - part. +- The [`ActResult`](#types) carries the step's `status`, its `summary`, the + `modelCalls` and `actions` it spent, and `cache`: how the + [replay cache](/cache) took part. `status` is `done`, or `stopped` when an + [`@e2e-dev/core`](/core#keep-actions-for-your-code) agent's guard stopped the + step before the action in `stoppedAt`. - The agent acts through the [action tools](#action-tools) and requests screenshots itself through the [pixel tools](#pixel-tools-screenshot-and-the-point-verbs). @@ -658,8 +660,19 @@ export interface VisionOption { */ export type VisionMode = boolean | 'only'; -/** What one passing `act` step did, as the report records it. */ -export interface ActResult { +/** + * What one passing `act` step did, as the report records it. `status` is + * `done` when the step finished, and `stopped` when an `@e2e-dev/core` agent's + * guard stopped it before an action, for the program to take over; only a + * guard stops a step, and `stoppedAt` is the action it stopped before. + */ +export type ActResult = ActOutcome & ( + | { readonly status: 'done'; readonly stoppedAt?: never } + | { readonly status: 'stopped'; readonly stoppedAt: ProposedAction } +); + +/** What every passing `act` step reports, whatever its `status`. */ +export interface ActOutcome { /** The executor's one-line account of the step, or the replay's when the cache finished it. */ readonly summary: string; /** How the replay cache took part; absent when caching is off for the step. */ diff --git a/docs/reference/app.mdx b/docs/reference/app.mdx index 182230d1b1..ef3fe31253 100644 --- a/docs/reference/app.mdx +++ b/docs/reference/app.mdx @@ -64,13 +64,16 @@ await app.open('/settings/billing'); // a path relative to app.url await app.open('https://example.com/login'); // any http(s) URL ``` -On a device target, which has no URL, `open()` launches the pinned app fresh -and takes no path. +On a device target, which has no URL, `open()` launches the pinned app fresh. +A path opens as a link in the app when the target declares +[`app.linkBase`](/reference/config#linkbase), and is `APP_URL_REQUIRED` +without it. With a link base, an absolute link opens as given, custom app +schemes included. | Throws | When | | --- | --- | -| `APP_URL_REQUIRED` | A path, on a target that declares no `app.url`. | -| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:`. `about:blank` is allowed. | +| `APP_URL_REQUIRED` | A path, on a target that declares no `app.url` (or, on a device, no `app.linkBase`). | +| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:`. `about:blank` is allowed. On a device with `app.linkBase`, a link whose scheme is `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, or `filesystem:`. | | `APP_UNREACHABLE` | The app URL could not be reached. | | `ACTION_FAILED` | The operation failed or timed out. | | `UNSUPPORTED_CAPABILITY` | No `session.open` on a target with a URL, or no `session.restart` on one without. On a device with no `app.bundleId`, also the target's build not installed yet ([mobile](/reference/mobile#the-app-under-test)). | @@ -211,7 +214,8 @@ export interface App { /** * Opens the app: the declared URL, a path relative to it, or any absolute * http(s) URL. On a device target, which has no URL, it launches the pinned - * app fresh and takes no path. + * app fresh; a path opens as a link in it when the target declares + * `app.linkBase`. */ open(path?: string): Promise; /** diff --git a/docs/reference/config.mdx b/docs/reference/config.mdx index 141144797f..73bdfe65a5 100644 --- a/docs/reference/config.mdx +++ b/docs/reference/config.mdx @@ -353,6 +353,7 @@ targets: [ | [`url`](#url) | `string` | - | | [`bundleId`](#bundleid) | `string` | - | | [`appPath`](#apppath) | `string` | - | +| [`linkBase`](#linkbase) | `string` | - | | [`identity`](#identity) | `string` | derived from the app | | [`environment`](#environment) | `'test' \| 'staging' \| 'production'` | `'test'` or `'production'` | | [`launchArguments`](#launcharguments) | `string[]` | - | @@ -367,6 +368,7 @@ target. A target with `app` needs an [engine](#engine). `launchArguments`. - A `mobile()` target needs `app.bundleId` or `app.appPath`. `app.url` on a mobile target is not supported yet. +- `app.linkBase` needs an engine that opens links, such as `mobile()`. - An unknown key in `app` or `app.command` fails with a did-you-mean. @@ -420,6 +422,29 @@ appPath: './build/App.app', Without `bundleId`, the app it installs is the one launched. See [the device engine](/reference/mobile#engine) for when it is installed. +#### `linkBase` + +Where `app.open(path)` opens on a device target: the path joins the link base, +and the link opens in the pinned app. + +```ts +linkBase: 'https://shop.example.com', // app.open('/orders/42') opens https://shop.example.com/orders/42 +linkBase: 'shopapp://', // app.open('/orders/42') opens shopapp://orders/42 +``` + +- One slash joins the base and the path; none follows a bare scheme + (`shopapp://`). +- An absolute link passed to `app.open` opens as given, a custom app scheme + included; `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, and + `filesystem:` links are `POLICY_DENIED`. +- Without `linkBase`, `app.open()` on a device launches the app and takes no + path (`APP_URL_REQUIRED`). + +| Throws | When | +| --- | --- | +| `INVALID_CONFIG` | The engine opens no links (a `web()` target), the value is not an absolute URL or a scheme, it has a query or fragment (an empty `?` or `#` included), it has leading or trailing whitespace, or it names `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, or `filesystem:`. | +| `POLICY_DENIED` | `app.open` got an absolute link with one of those schemes. | + #### `identity` The stable identity keying replay cache and session entries. diff --git a/docs/reference/engine-protocol.mdx b/docs/reference/engine-protocol.mdx index 7fc0a1c75b..5848148d93 100644 --- a/docs/reference/engine-protocol.mdx +++ b/docs/reference/engine-protocol.mdx @@ -231,6 +231,7 @@ params are the member's arguments by name, with every `OperationContext` as | `artifacts/startVideo` | `{ operation }` | `null` | | `artifacts/stopVideo` | `{ operation }` | `VideoSegment[]` | | `session/open` | `{ url, operation }` | `null` | +| `session/openLink` | `{ url, operation }` | `null` | | `session/back`, `session/restart`, `session/reset` | `{ operation }` | `null` | | `session/resize` | `{ size: { width, height }, operation }` | `null` | | `session/switchTab` | `{ index, operation }` | `null` | diff --git a/docs/reference/engine.mdx b/docs/reference/engine.mdx index 22c01dd470..4f4aac0e2f 100644 --- a/docs/reference/engine.mdx +++ b/docs/reference/engine.mdx @@ -12,8 +12,9 @@ implementation, derives its capabilities, and returns a handle for `targets` in the config. The runner drives it over the [engine protocol](/reference/engine-protocol), in its own process; see [Engines](/engines). Use `test.extend` fixtures for resources outside the UI, -such as APIs or databases. `@e2e-dev/core/engine` is the same module, and a -published engine imports it from there and peers on `@e2e-dev/core`. +such as APIs or databases. `@e2e-dev/core/engine` is the same module: a +published engine imports it from there and peers on `@e2e-dev/core`, so it runs +under the runner and under [@e2e-dev/core](/core) alike. This sketch assumes a `connect` adapter for the platform: @@ -67,11 +68,11 @@ Validates an engine spec, freezes it, computes its ```ts import { defineEngine, isEngineHandle, type EngineHandle } from 'e2e/engine'; -export function defineEngine(spec: Engine): EngineHandle; +export function defineEngine(spec: Engine): EngineHandle; export function isEngineHandle(value: unknown): value is EngineHandle; /** A validated engine: branded, frozen, capabilities computed. */ -export interface EngineHandle extends Engine { +export interface EngineHandle extends Engine { readonly capabilities: ReadonlySet; } ``` @@ -79,6 +80,12 @@ export interface EngineHandle extends Engine { Every function on the handle is bound to the spec it came from, so a class instance is as valid a body as an object literal. +`Fixtures` types what the engine's [`fixtures`](#contributed-fixtures) contribute, such as +`{ browser: Browser }`. It is a type only, never checked against `fixtures` at +runtime. With it, `launch` from [`@e2e-dev/core`](/core) types a session's fixtures +without a hand-written argument: return `defineEngine<{ device: Device }>(...)` +from the factory. + `defineEngine` throws `INVALID_CONFIG` when the spec breaks one of these rules: @@ -463,7 +470,7 @@ engine. | `test.setup` sessions, the `session:` option | `state` | | `app.screenshot()`, assertion evidence, the failure screenshot | `artifacts` | | every [`video` mode](/reference/config#recording) but `off`, from the config, a target, a test, or `--video` | `artifacts` | -| `app.open()` | `session.open`; on a target with no app URL, `session.restart` | +| `app.open()` | `session.open`; on a target with no app URL, `session.restart`, or `session.openLink` for a path on a target with `app.linkBase` | | `app.back()` | `session.back` | | `app.restart()` | `session.restart` | | `app.clearState()` | `session.reset` | @@ -1433,6 +1440,7 @@ Steering hooks behind the universal `app` fixture and the agent's `navigate`, | Hook | Behind | | --- | --- | | `open(url)` | `app.open()`, the agent's `navigate` verb, the reopen after `restart` and `reset` | +| `openLink(url)` | `app.open(path)` on a target with `app.linkBase`: the joined link, its scheme checked, opened in the app under test. A target can declare `linkBase` only when its engine has this hook. | | `back()` | `app.back()`, the agent's `back` verb | | `resize(size)` | the agent's `resize_viewport` verb; `size` is `{ width, height }` in CSS pixels and holds for the rest of the attempt | | `switchTab(index)` | the agent's `switch_tab` verb; `index` is zero-based in the order the tabs opened, as [`tabs`](#observation) counts them. An index with no open tab is `ACTION_FAILED`. | @@ -1467,6 +1475,13 @@ export interface EngineSession { * origin policy. Absent on a surface without addressable locations. */ open?(url: string, context: OperationContext): Promise; + /** + * Opens a link in the app under test, as a deep link or a universal link + * would: `app.open(path)` on a target that declares `app.linkBase` lands + * here with the link joined and its scheme checked. A device engine's hook; + * a target whose engine lacks it cannot declare `linkBase`. + */ + openLink?(url: string, context: OperationContext): Promise; /** Navigates back once in the surface's history. */ back?(context: OperationContext): Promise; /** Sets the viewport to `size` in CSS pixels, for this page and every page the attempt opens after it. */ @@ -1613,6 +1628,7 @@ reproduces these semantics; import them instead of carrying a copy. | [`resolveExpression`](#resolveexpression) | `SemanticNode[]` | | [`urlMatches`](#urlmatches) | `boolean` | | [`siteOf`, `sameSite`](#site-helpers) | `string`, `boolean` | +| [`isRefusedLinkScheme`](#isrefusedlinkscheme) | `boolean` | | [`pollCondition`, `Deadline`](#pollcondition) | `Promise`, `Deadline` | | [`validateJsonValue`](#validatejsonvalue) | `void` | | [`rejectUnknownKeys`](#rejectunknownkeys) | `void` | @@ -1703,6 +1719,21 @@ export function siteOf(hostname: string): string; export function sameSite(url: string | URL, site: string): boolean; ``` +#### `isRefusedLinkScheme` + +Whether a link's scheme is one no app link may take: `file:`, `data:`, +`javascript:`, `view-source:`, `blob:`, or `filesystem:`. An engine that +opens app links (`session.openLink`) refuses these with `POLICY_DENIED`, as +`app.open` does with `app.linkBase`. + +```ts +export function isRefusedLinkScheme(protocol: string): boolean; +``` + +```ts +if (isRefusedLinkScheme(new URL(link).protocol)) throw new ConfigurationError('POLICY_DENIED', 'refused link scheme'); +``` + #### `pollCondition` Assertion polling and the negation grace window. diff --git a/docs/reference/errors.mdx b/docs/reference/errors.mdx index 8dcc3e9139..f4fe48512c 100644 --- a/docs/reference/errors.mdx +++ b/docs/reference/errors.mdx @@ -95,13 +95,13 @@ category a blocked verdict reports as the obstacle. | --- | --- | --- | ---: | | `AUTH_CREDENTIAL_UNAVAILABLE` | A `Secret` or `credentials.user()` names a credential missing from `config.credentials`, the call ran outside an active runner, or the password's provider returned no string of at least 6 characters (code points) | configuration | 2 | | `AUTH_CREDENTIAL_INVALID` | The application rejected a configured credential (blocked: credentials) | configuration | 2 | -| `SECRET_UNAVAILABLE` | A `Secret` or `secrets.get()` during a run names a secret missing from `config.secrets`, the secret's provider returned no string of at least 6 characters (code points), or an engine resolved a secret it did not declare in its `secrets` | configuration | 2 | +| `SECRET_UNAVAILABLE` | A `Secret` or `secrets.get()` during a run names a secret missing from `config.secrets`, the secret's provider returned no string of at least 6 characters (code points), or an engine resolved a secret it did not declare in its `secrets`. Also `host.secret(name)` (`@e2e-dev/core`) naming a secret not passed to `launch` | configuration | 2 | | `ENVIRONMENT_UNAVAILABLE` | The environment under test is down or unreachable beyond the app itself (blocked: environment) | infrastructure | 3 | | `SEED_DATA_MISSING` | Data the step depends on does not exist in the environment (blocked: seed_data) | configuration | 2 | | `TEST_SETUP_FAILED` | Test prerequisites outside the product failed to arrange (blocked: test_setup). Also a `test.extend()` fixture that broke its contract: it returned without calling `use()`, called it twice, or redefined a fixture the target's engine contributes | configuration | 2 | | `AUTOMATION_UNSUPPORTED` | The step needs an interaction the available toolset cannot perform, such as a right-click on a device (blocked: automation) | test | 1 | | `MODEL_UNAVAILABLE` | No model configured: the built-in agent's `agents..model` holds no AI SDK model instance (the message names the selected agent), or GitHub Copilot does not serve the configured model to this account (`model_not_supported`; the message names the model). Recorded once as a run-level error, and the run stops. In a test run, a missing model is found when the `agent` fixture is first acquired, and an unserved model at its first call. `e2e explore` runs the built-in agent, so it checks for a missing model before anything starts, a custom executor's entry included. A custom executor without a model is not checked: its `act` and `assert` steps run, and `waitFor` or `extract` fails with this code on its first call. Evicts no [recording](/cache). | configuration | 2 | -| `POLICY_DENIED` | The security policy refused an operation: a denied destination, a forbidden fill, `vision: 'only'` without pixels, or a read from a secure field. [Policy denials](/reference/security#policy-denials) lists every case. An unknown tool or an undeclared field goes back to the model as the call's result, never as this code | configuration | 2 | +| `POLICY_DENIED` | The security policy refused an operation: a denied destination, a forbidden fill, `vision: 'only'` without pixels, or a read from a secure field. Also an action an `@e2e-dev/core` agent's `guard` denies, which the model reads as the action's result, and an `app.open` link with a refused scheme. [Policy denials](/reference/security#policy-denials) lists every case. An unknown tool or an undeclared field goes back to the model as the call's result, never as this code | configuration | 2 | | `APP_UNREACHABLE` | The app URL could not be reached, or an app command exited or exceeded `startupTimeout`. An exit after readiness during `e2e run` interrupts active tests and skips queued tests only for dependent targets with unfinished work. See [app commands](/reference/config#command) for diagnostics. Under the agent, also any engine fault with no more specific code. On a device, the message can identify a busy or wedged iOS automation runner and its recovery. See [Mobile](/mobile#troubleshooting). | infrastructure | 3 | | `APP_ALREADY_RUNNING` | `readyUrl` answered before the target's `app.command` was spawned, and `reuseExisting` was not in effect. Stop that process, or set `reuseExisting: true` locally | infrastructure | 3 | | `MODEL_PROVIDER_FAILED` | Provider transport failure (network error, 5xx, rate limit, or no response within 120 s) after the adapter's transport retries: up to 5 per model call, honoring `retry-after`, within the remaining step timeout. A stalled request is aborted at 120 s and sent again; an `agent.act` step's `--debug` transcript notes it. The 120 s covers the whole request, so a model that needs longer for one answer fails here too. An error body the provider package cannot read shows as the HTTP status and the body, secrets redacted, cut at 1 KB. Also a call the provider ended on an `error` finish, whatever text came with it, and an `e2e doctor --check-model` call with no answer within 1 minute. Evicts no [recording](/cache) | infrastructure | 3 | @@ -130,9 +130,9 @@ These appear in terminal output and in the report as plain string codes. | `CONFIG_NOT_FOUND` | `--config ` names a file that does not exist, or no `e2e.config.ts`, `e2e.config.mts`, `e2e.config.js`, or `e2e.config.mjs` exists from the working directory up to the repository root | configuration | 2 | | `CONFIG_AMBIGUOUS` | Both `e2e.config.ts` and `e2e.config.mts` in one directory, or both `e2e.config.js` and `e2e.config.mjs` in one with no TypeScript config | configuration | 2 | | `CONFIG_LOAD_FAILED` | The config module failed to import: a package that is declared but not installed, an undeclared package, a wrong subpath, a removed export, or any other throw while it evaluates. The message names the fix. A config error raised while it evaluates keeps its own code: options a factory the config calls refuses, or a `secrets.get()` handle turned into a string, is `INVALID_CONFIG` | configuration | 2 | -| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`, `decisionExecutor()`, `decisionFilter()`) refused its options, with the factory's own message. Or: an unknown key, out-of-range value, invalid enum, or missing `targets`; a target key outside `{ name, platform, engine, app, trace, video }`; a target `app` key that is unknown or malformed; `app` on a target without an engine; a `web()` target without `app.url` or with a device field; a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), with a link in `bundleId`, or with an unknown permission name or state (the message names the nearest name); an unknown key in a `web()` or `mobile()` option, `web({ connect })` and `web({ basicAuth })` included (the message names the nearest one); a `trace` or `video` that is not a recording mode; the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) or `artifacts.trace` block (the message names the `trace` mode they meant); the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option); the removed `specVersion` key; a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one); a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`), or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one); an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead); an unknown key in `app.command` (the message names the nearest one); an `engine` that is not a `defineEngine` handle; the `json` reporter combined with `list`, or in a run with `--step`; a duplicate or malformed target name; a secret or credential password shorter than 6 characters (code points); a credential that is not `{ username, password }`, whose `username` is not a string (only `password` may be a function), or that has no username in the config or `E2E_USER__USERNAME`; two credentials or two secrets whose names map to one `E2E_USER_*` or `E2E_SECRET_*` override variable (`api-key` and `api_key`); an oversized `agents..context`; an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, or a `timeout` of 0 outside a test; a `tests` value that is not a glob or a list of globs, or a `tests` list of only `!` exclusions; the removed top-level `limits` key; an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`); an unknown agent key (the message names the nearest one); the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`); `system` or `tools` beside `executor`; a `tools` entry that `defineTool` did not build, or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `run_code`, `test_lines`, `start_recording`, `stop_recording`, or `report_finding`); `createToolLoopExecutor({ maxTurns })`; `reuseExisting` on an app whose `app.url` asks for a free port; a port-0 `app.url` with no `app.command` to serve it; a `filters` value that is not a list of `{ name, filter }` (for a bare function such as `decisionFilter`, the message says to call it); or a filter `name` that breaks the target name rules, is longer than 256 characters, or is used twice | configuration | 2 | +| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`, `decisionExecutor()`, `decisionFilter()`) refused its options, with the factory's own message. Or: an unknown key, out-of-range value, invalid enum, or missing `targets`; a target key outside `{ name, platform, engine, app, trace, video }`; a target `app` key that is unknown or malformed; `app` on a target without an engine; a `web()` target without `app.url` or with a device field; a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), an `app.linkBase` on a target whose engine opens no links, or one that is not an absolute URL or scheme, has a query or fragment, or names a refused scheme, with a link in `bundleId`, or with an unknown permission name or state (the message names the nearest name); an unknown key in a `web()` or `mobile()` option, `web({ connect })` and `web({ basicAuth })` included (the message names the nearest one); a `trace` or `video` that is not a recording mode; the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) or `artifacts.trace` block (the message names the `trace` mode they meant); the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option); the removed `specVersion` key; a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one); a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`), or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one); an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead); an unknown key in `app.command` (the message names the nearest one); an `engine` that is not a `defineEngine` handle; the `json` reporter combined with `list`, or in a run with `--step`; a duplicate or malformed target name; a secret or credential password shorter than 6 characters (code points); a credential that is not `{ username, password }`, whose `username` is not a string (only `password` may be a function), or that has no username in the config or `E2E_USER__USERNAME`; two credentials or two secrets whose names map to one `E2E_USER_*` or `E2E_SECRET_*` override variable (`api-key` and `api_key`); an oversized `agents..context`; an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, or a `timeout` of 0 outside a test; a `tests` value that is not a glob or a list of globs, or a `tests` list of only `!` exclusions; the removed top-level `limits` key; an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`); an unknown agent key (the message names the nearest one); the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`); `system` or `tools` beside `executor`; a `tools` entry that `defineTool` did not build, or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `run_code`, `test_lines`, `start_recording`, `stop_recording`, or `report_finding`); `createToolLoopExecutor({ maxTurns })`; `reuseExisting` on an app whose `app.url` asks for a free port; a port-0 `app.url` with no `app.command` to serve it; a `filters` value that is not a list of `{ name, filter }` (for a bare function such as `decisionFilter`, the message says to call it); or a filter `name` that breaks the target name rules, is longer than 256 characters, or is used twice | configuration | 2 | | `CONFIG_NOT_DETERMINISTIC` | A worker re-resolved the config module and got a different digest than the runner | configuration | 2 | -| `APP_URL_REQUIRED` | A navigation or relative URL on a target that declares no `app.url`, or an `app.command` with no `app.url` or `app.readyUrl` to poll | configuration | 2 | +| `APP_URL_REQUIRED` | A navigation or relative URL on a target that declares no `app.url` (on a device target with no `app.linkBase`, `app.open(path)`), or an `app.command` with no `app.url` or `app.readyUrl` to poll | configuration | 2 | | `INVALID_APP_URL` | A base URL that is unparseable or has userinfo, a query, a fragment, a non-HTTP scheme, or plain HTTP on a non-loopback host; port 0 on a host other than a literal loopback address (`127.0.0.1`, `[::1]`); or a loopback address the run could not bind a free port on | configuration | 2 | | `INVALID_GLOB` | An empty pattern; a first segment that starts with `!` beyond the one leading `!` of a `tests` exclusion (`!!tests/**`, `./!tests/**`); a bare `!`; braces, character classes, or extglobs; a backslash separator; an absolute path or a `..` segment; a trailing `/` or `.`; a `tests` entry with no wildcard that names a directory (the message names the glob to write); a pattern that names only the project root; or a `**` that is not a whole segment | configuration | 2 | | `UNKNOWN_TARGET` | `--target`, `open_session.target`, or `openHost` (`e2e/runner`) named a target that is not configured | configuration | 2 | @@ -155,9 +155,9 @@ These appear in terminal output and in the report as plain string codes. | `INVALID_CHANGED_SINCE` | `--changed-since` (`changedSince`) is not a string, is empty, or starts with `-`. Or, when the run calls test filters or `e2e explore` explores the change, it names no commit in the checkout or shares no history with HEAD (in a shallow clone the message says to fetch more history, otherwise that the histories are unrelated); or the project is not in a git repository, git is missing, or git did not answer within 30 seconds | configuration | 2 | | `ONLY_IN_CI` | A focused test reached a CI run | configuration | 2 | | `COLLECTION_ERROR` | Registration outside collection; an invalid title; a non-function body; an async `describe` body; nested serial groups; an option forbidden inside a serial group; a setup test inside `describe`; an invalid session name; a malformed test option, such as `tags` that is not a list of distinct names; a test or `describe` option key the call does not take (the message names the nearest one); a test file that fails to import, in a run with no positionals or when the positionals selected it; a session that a selected test consumes, or `e2e explore --session` names, but no collected setup test produces (the message lists the declared sessions); a consumed session on a target whose engine has no `state` capability to save it; or a positional path outside the project root | configuration | 2 | -| `SESSION_MISMATCH` | A stored session does not match the current run, target, engine, platform, or app identity | configuration | 2 | -| `SESSION_EXPIRED` | A stored session is past its expiry at restore time (configuration, exit 2), or a setup test saved state whose expiry is not after its creation (test, exit 1) | configuration or test | 2 or 1 | -| `SESSION_INVALID` | A stored session failed authentication. Tampered or corrupt. | configuration | 2 | +| `SESSION_MISMATCH` | A stored session does not match the current run, target, engine, platform, or app identity. Also an `@e2e-dev/core` `SessionState` passed to a host of another engine, platform, or app | configuration | 2 | +| `SESSION_EXPIRED` | A stored session, or an `@e2e-dev/core` `SessionState`, is past its expiry at restore or import time (configuration, exit 2), or a setup test saved state whose expiry is not after its creation (test, exit 1) | configuration or test | 2 or 1 | +| `SESSION_INVALID` | A stored session failed authentication. Tampered or corrupt. Also `SessionState.import` (`@e2e-dev/core`) on bytes that are not a `session-state-1` document or that fail authentication under the key: another key, or a changed byte | configuration | 2 | | `SESSION_UNAVAILABLE` | A test requires a session no selected setup test produced | configuration | 2 | | `SESSION_CONTRACT` | A setup test saved a session twice, saved an undeclared name, or finished without saving a declared one | test | 1 | | `TEST_TIMEOUT` | The attempt exceeded the resolved test `timeout`. The body is not stopped: the harness operation it is in is cancelled, and its fixtures die with the attempt, so it cannot reach the next test. A test that blocks the event loop gets its worker killed once `cleanupTimeout` (at least 5 s) passes after the timeout. See [Hooks](/reference/test#hooks) | test | 1 | @@ -177,7 +177,7 @@ These appear in terminal output and in the report as plain string codes. | `TYPESCRIPT_COMPILER_UNAVAILABLE` | oxc, which compiles e2e's TypeScript, could not be loaded; the message gives the cause. Most often its native bindings for this platform (`@oxc-transform/binding--`, `@oxc-parser/binding--`) are not installed. npm leaves them out with `--omit=optional`, or when `package-lock.json` was written on another platform (npm/cli#4828). Install again without `--omit=optional`, or delete `package-lock.json` and `node_modules` and run `npm install` | infrastructure | 3 | | `IMAGE_CODEC_UNAVAILABLE` | `@napi-rs/image`, which reads and writes the PNGs of screenshots, could not be loaded; the message gives the cause. It is loaded when the first screenshot is read or written, so a run that takes none never needs it. Most often its native binding for this platform (`@napi-rs/image--*`) is not installed. npm leaves it out with `--omit=optional`, or when `package-lock.json` was written on another platform (npm/cli#4828). Install again without `--omit=optional`, or delete `package-lock.json` and `node_modules` and run `npm install`. The package publishes no binding for Linux on ppc64, s390x, or riscv64, where `toHaveScreenshot` cannot work, and neither can an agent screenshot the runner has to resize (one with a side over 768 px) | infrastructure | 3 | | `ENGINE_FAILURE` | Any engine fault with no more specific mapping | infrastructure | 3 | -| `WORKER_PROTOCOL` | A worker reported the wrong unit, an attempt overlapped a live attempt, `openHost` tried to acquire an engine instance another host owns, or `host.openSession` tried to reuse a closed host or overlap an opening, live, or closing session | infrastructure | 3 | +| `WORKER_PROTOCOL` | A worker reported the wrong unit, an attempt overlapped a live attempt, `openHost` tried to acquire an engine instance another host owns, or `host.openSession` tried to reuse a closed host or overlap an opening, live, or closing session. In `@e2e-dev/core`, `launch` on an engine instance another host owns, or `host.newSession` on a closed host or while a session is open | infrastructure | 3 | | `SESSION_KEY_INVALID` | A worker received a session key that is not 32 bytes. Packaging or IPC fault. | infrastructure | 3 | | `REPORT_WRITE_FAILED` | `report.json` or `ai-trace.json` could not be written. Recorded as a run error; the in-memory outcome is still complete. A `junit.xml` that could not be written is a line on the `junit` reporter's stderr instead | infrastructure | 3 | | `EVALUATE_FAILED` | `browser.evaluate` threw inside the page, or the page deleted or replaced a JavaScript built-in (such as `Map`) that the engine's in-page code needs, which fails later evaluates and locators (`@e2e-dev/web`) | test | 1 | diff --git a/docs/reference/mobile.mdx b/docs/reference/mobile.mdx index 75f3fbf9e8..101a45599a 100644 --- a/docs/reference/mobile.mdx +++ b/docs/reference/mobile.mdx @@ -295,8 +295,9 @@ export default { - A device target needs `app.bundleId` or `app.appPath`. - `app.open()` launches the target's app fresh. An attempt launches nothing on its own, so a test starts where the previous one left the app. -- `app.open` takes no path on a device: a link goes through - [`device.openLink`](#openlink). +- `app.open(path)` opens the path as a link in the app when the target + declares [`app.linkBase`](/reference/config#linkbase); without it, `app.open` + takes no path and a link goes through [`device.openLink`](#openlink). - `app.appPath` is the build the suite runs against. The engine installs nothing on its own: [`device.installApp()`](#installapp) installs it where the suite says so. A device provider installs it on the device it leases. @@ -411,7 +412,7 @@ them to finish. | `keyboard` | `type` and `press` into the focused field, the agent's `type` and `press` without a target and the fallback of `type_at` and `press_at`, and `dismiss`, the agent's `dismiss_keyboard`. | | `locate` | Every `screen` query. Text and label queries answer with the innermost match, since a device echoes both up the tree (a React Native input is a host view and the field inside it, both labelled). On Android a test id query, a string or a RegExp, also matches a resource id without its `:id/` prefix, for any package but the framework's `android`, when nothing on the screen matches the full id. `screen.locator('id=…')` is agent-device's selector and takes the full id. | | `validateApp` | Requires `app.bundleId` or `app.appPath` and refuses `app.url` for now, with a note on reaching this machine from the platform's device. A target with either is a `native-app` target, so `requires: ['native-app']` selects device targets. | -| `session` | `back`, `restart`, and `reset`. No `open`: `app.open()` takes no path and the runner serves it with `restart`, the fresh launch of the pinned app; the agent's `navigate` verb is unavailable, and `device.openLink(url)` opens a deep link or web link. | +| `session` | `back`, `restart`, `reset`, and `openLink`, behind `app.open(path)` on a target with `app.linkBase`. No `open`: `app.open()` with no path is served with `restart`, the fresh launch of the pinned app; the agent's `navigate` verb is unavailable, and `device.openLink(url)` opens a deep link or web link. | | `artifacts` | Redacted screenshots under `screenshots/NNN-